[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:48 dist-test-slave-mr2f krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:48 dist-test-slave-mr2f krb5kdc[399](info): set up 2 sockets
Sep 24 05:08:48 dist-test-slave-mr2f krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 05:08:49.902076 392 mini_kdc.cc:121] Time spent starting KDC: real 2.143s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 05:08:50.061160 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.159s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 05:08:50.236270 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.110s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/test-user.keytab.
W20260924 05:08:51.148384 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.912s user 0.000s sys 0.002s
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:51.190311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:51.269701 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:51.269912 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:51.269955 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:51.271114 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:51.271171 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:51.271198 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:51.271217 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:51.271245 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:51.272606 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:51.272878 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:51.273113 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:51.278441 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:51.278514 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:51.278569 412 server_base.cc:1034] running on GCE node
W20260924 05:08:51.278489 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:51.278968 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:51.279249 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:51.280381 412 hybrid_clock.cc:648] HybridClock initialized: now 1790226531280366 us; error 36 us; skew 500 ppm
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:51.282895 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:51.283334 412 webserver.cc:460] Webserver started at http://127.0.98.62:33577/ using document root <none> and password file <none>
I20260924 05:08:51.283526 412 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:51.283566 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:51.283654 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:51.284616 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "0a2f93d56fdab030c2b754ff22c42698"
server_key_iv: "080cf6d04a4c5a725246f15b1ca9b341"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.284916 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "0a2f93d56fdab030c2b754ff22c42698"
server_key_iv: "080cf6d04a4c5a725246f15b1ca9b341"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.286165 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:08:51.286846 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.287007 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:51.287081 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "0a2f93d56fdab030c2b754ff22c42698"
server_key_iv: "080cf6d04a4c5a725246f15b1ca9b341"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.287135 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:51.301149 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:51.302877 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:51.302995 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:51.306416 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34049
I20260924 05:08:51.306483 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34049 every 8 connection(s)
I20260924 05:08:51.306859 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:51.307796 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:51.309937 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0: Bootstrap starting.
I20260924 05:08:51.310482 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:51.310753 480 log.cc:824] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:51.311380 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0: No bootstrap required, opened a new log
I20260924 05:08:51.312899 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } }
I20260924 05:08:51.313047 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:51.313092 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6613c8b7bd948b99d1af7a3e7d71aa0, State: Initialized, Role: FOLLOWER
I20260924 05:08:51.313241 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } }
I20260924 05:08:51.313333 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:08:51.313376 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:08:51.313424 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:51.313908 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } }
I20260924 05:08:51.314016 480 leader_election.cc:302] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6613c8b7bd948b99d1af7a3e7d71aa0; no voters:
I20260924 05:08:51.314179 480 leader_election.cc:288] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:08:51.314213 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:51.314375 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [term 1 LEADER]: Becoming Leader. State: Replica: d6613c8b7bd948b99d1af7a3e7d71aa0, State: Running, Role: LEADER
I20260924 05:08:51.314419 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:51.314453 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } }
I20260924 05:08:51.315012 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6613c8b7bd948b99d1af7a3e7d71aa0. Latest consensus state: current_term: 1 leader_uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } }
I20260924 05:08:51.315063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 412
I20260924 05:08:51.315100 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:51.315171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:51.315558 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:51.315973 392 external_mini_cluster.cc:1428] Setting key 2005b9ff45f09a1ae89d7ed508ee0cb2
I20260924 05:08:51.316043 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:51.315266 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6613c8b7bd948b99d1af7a3e7d71aa0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } }
I20260924 05:08:51.316890 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:51.317500 492 catalog_manager.cc:1347] Generated new cluster ID: cd8c80f39d2943d3a86a63d314379455
I20260924 05:08:51.317562 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:51.330152 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:51.330856 492 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:51.333729 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:51.316803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.0.98.62:34049)
0924 05:08:51.318885 (+ 2082us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:51.318895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:51.318930 (+ 35us) server_negotiation.cc:407] Connection header received
0924 05:08:51.319353 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:51.319360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:51.319415 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:51.319555 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:51.321434 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.323397 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.324787 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.325419 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.325525 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:51.329529 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:51.329533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:51.329534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:51.329545 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:51.331563 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.332079 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.332082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.332083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.332201 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.332504 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.332505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.332506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.332771 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:51.332957 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:51.333195 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 05:08:51.333342 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2038,"thread_start_us":71,"threads_started":1}
I20260924 05:08:51.335968 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6613c8b7bd948b99d1af7a3e7d71aa0: Generated new TSK 0
I20260924 05:08:51.336127 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:08:51.363528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:51.441982 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:51.442200 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:51.442224 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:51.443383 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:51.443426 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:51.443452 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:08:51.444864 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:51.445091 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:51.445317 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:51.450183 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:51.450192 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:51.450340 506 server_base.cc:1034] running on GCE node
W20260924 05:08:51.450202 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:51.450615 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:51.450896 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:51.452055 506 hybrid_clock.cc:648] HybridClock initialized: now 1790226531452045 us; error 56 us; skew 500 ppm
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:51.454449 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:08:51.454879 506 webserver.cc:460] Webserver started at http://127.0.98.1:36121/ using document root <none> and password file <none>
I20260924 05:08:51.455122 506 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:51.455163 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:51.455248 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:51.456214 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1d2fb6033824795b798ad2dcad927f3"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "9412fe741e047f1c2f440b925c20b3e4"
server_key_iv: "1c4a36feb34c36e380f909bc614250b0"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.456497 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1d2fb6033824795b798ad2dcad927f3"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "9412fe741e047f1c2f440b925c20b3e4"
server_key_iv: "1c4a36feb34c36e380f909bc614250b0"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.457719 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:08:51.458407 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.458592 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:08:51.458675 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "c1d2fb6033824795b798ad2dcad927f3"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "9412fe741e047f1c2f440b925c20b3e4"
server_key_iv: "1c4a36feb34c36e380f909bc614250b0"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.458724 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:51.471959 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:51.473636 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:51.473742 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:51.473902 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:08:51.474149 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:51.474181 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.474202 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:51.474215 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.479857 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41959
I20260924 05:08:51.479930 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41959 every 8 connection(s)
I20260924 05:08:51.480219 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:51.487833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 506
I20260924 05:08:51.487905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:08:51.488055 392 external_mini_cluster.cc:1428] Setting key be38d45e342e5536056e21b8760a99ce
I20260924 05:08:51.489184 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:51.480930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41983 (local address 127.0.98.62:34049)
0924 05:08:51.480983 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:51.480986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:51.481198 (+ 212us) server_negotiation.cc:407] Connection header received
0924 05:08:51.481637 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:51.481638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:51.481648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:51.481702 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:51.482556 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.483143 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.484079 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.484269 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.484325 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:51.486345 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:51.486346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:51.486346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:51.486352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:51.487588 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.488142 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.488143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.488144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.488209 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.488518 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.488519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.488519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.488653 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:51.488729 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:51.488982 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 05:08:51.489098 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:08:51.489336 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:51.480917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34049 (local address 127.0.98.1:41983)
0924 05:08:51.481084 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:51.481119 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:51.481559 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:51.481754 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:51.481757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:51.482301 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:51.482482 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:51.482486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.483220 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.483221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:51.484008 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:51.484010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.484340 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.484340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:51.484432 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:51.484930 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:51.484939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:51.486244 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:51.487658 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:51.487663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:51.487665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:51.488008 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:51.488315 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:51.488316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:51.488317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:51.488447 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:51.488720 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:51.488726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:51.488856 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 05:08:51.488958 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 05:08:51.489108 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260924 05:08:51.489804 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34049
I20260924 05:08:51.489904 635 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:51.490100 635 heartbeater.cc:511] Master 127.0.98.62:34049 requested a full tablet report, sending...
I20260924 05:08:51.490892 444 ts_manager.cc:194] Registered new tserver with Master: c1d2fb6033824795b798ad2dcad927f3 (127.0.98.1:41959)
I20260924 05:08:51.491721 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41983
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:08:51.512961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:51.591148 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:51.591327 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:51.591349 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:51.592432 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:51.592473 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:51.592528 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:08:51.593962 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:51.594208 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:51.594441 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:51.599314 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:51.599352 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:51.599426 640 server_base.cc:1034] running on GCE node
W20260924 05:08:51.599570 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:51.599792 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:51.599977 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:51.601143 640 hybrid_clock.cc:648] HybridClock initialized: now 1790226531601124 us; error 32 us; skew 500 ppm
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:51.603520 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:08:51.603942 640 webserver.cc:460] Webserver started at http://127.0.98.2:45299/ using document root <none> and password file <none>
I20260924 05:08:51.604144 640 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:51.604183 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:51.604264 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:51.605193 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "ec1ce0ecb2754e60a51adf9563c11ca3"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "78b68b3bfe170471f23d517634cf4db4"
server_key_iv: "8517ab5b4e6094b1c6d8869927ececad"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.605476 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ec1ce0ecb2754e60a51adf9563c11ca3"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "78b68b3bfe170471f23d517634cf4db4"
server_key_iv: "8517ab5b4e6094b1c6d8869927ececad"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.606669 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:08:51.607385 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.607549 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:51.607620 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "ec1ce0ecb2754e60a51adf9563c11ca3"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "78b68b3bfe170471f23d517634cf4db4"
server_key_iv: "8517ab5b4e6094b1c6d8869927ececad"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.607676 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:51.623885 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:51.626060 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:51.626186 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:51.626358 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:08:51.626690 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:51.626726 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.626748 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:51.626824 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.632403 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43545
I20260924 05:08:51.632471 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43545 every 8 connection(s)
I20260924 05:08:51.632767 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:08:51.637813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 640
I20260924 05:08:51.637885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:08:51.638041 392 external_mini_cluster.cc:1428] Setting key 529ca111d43d2e5bd8177b5c1ee5679e
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:51.642184 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:51.633460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34049 (local address 127.0.98.2:51105)
0924 05:08:51.633626 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:51.633658 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:51.634137 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:51.634379 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:51.634383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:51.635020 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:51.635210 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:51.635214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.635900 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.635902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:51.636698 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:51.636700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.637059 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.637060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:51.637175 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:51.637569 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:51.637577 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:51.639079 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:51.641002 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:51.641007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:51.641009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:51.641218 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:51.641409 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:51.641409 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:51.641410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:51.641496 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:51.641751 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:51.641754 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:51.641857 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 05:08:51.641932 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:08:51.642048 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260924 05:08:51.642187 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:51.633480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51105 (local address 127.0.98.62:34049)
0924 05:08:51.633532 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:51.633535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:51.633734 (+ 199us) server_negotiation.cc:407] Connection header received
0924 05:08:51.634245 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:51.634246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:51.634258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:51.634313 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:51.635307 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.635798 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.636784 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.636963 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.637042 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:51.639166 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:51.639168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:51.639168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:51.639174 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:51.640912 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.641288 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.641289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.641290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.641347 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.641543 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.641544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.641544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.641689 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:51.641757 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:51.641971 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 05:08:51.642067 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:08:51.642980 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34049
I20260924 05:08:51.643085 769 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:51.643245 769 heartbeater.cc:511] Master 127.0.98.62:34049 requested a full tablet report, sending...
I20260924 05:08:51.643599 444 ts_manager.cc:194] Registered new tserver with Master: ec1ce0ecb2754e60a51adf9563c11ca3 (127.0.98.2:43545)
I20260924 05:08:51.643954 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51105
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:08:51.661892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:51.740070 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:51.740290 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:51.740326 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:51.741446 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:51.741508 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:51.741552 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:08:51.743019 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:51.743264 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:51.743475 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:51.748562 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:51.748667 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:51.748590 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:51.748739 774 server_base.cc:1034] running on GCE node
I20260924 05:08:51.748905 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:51.749092 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:51.750222 774 hybrid_clock.cc:648] HybridClock initialized: now 1790226531750214 us; error 25 us; skew 500 ppm
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:51.752592 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:08:51.753000 774 webserver.cc:460] Webserver started at http://127.0.98.3:35863/ using document root <none> and password file <none>
I20260924 05:08:51.753181 774 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:51.753219 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:51.753300 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:51.754231 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "6270acc9a84548268f79c42fb68ae6e7"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "6b051c5cde6240da0a70a6d49823a633"
server_key_iv: "db0c3432a92070add32e0cb4c36692c3"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.754514 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6270acc9a84548268f79c42fb68ae6e7"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "6b051c5cde6240da0a70a6d49823a633"
server_key_iv: "db0c3432a92070add32e0cb4c36692c3"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.755759 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:08:51.756421 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.756567 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:08:51.756645 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "6270acc9a84548268f79c42fb68ae6e7"
format_stamp: "Formatted at 2026-09-24 05:08:51 on dist-test-slave-mr2f"
server_key: "6b051c5cde6240da0a70a6d49823a633"
server_key_iv: "db0c3432a92070add32e0cb4c36692c3"
server_key_version: "encryptionkey@0"
I20260924 05:08:51.756704 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:51.780545 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:51.782409 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:51.782521 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:51.782708 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:08:51.783042 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:51.783079 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.783133 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:51.783165 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:51.789745 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40247
I20260924 05:08:51.789819 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40247 every 8 connection(s)
I20260924 05:08:51.790103 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:51.796411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 774
I20260924 05:08:51.796484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:08:51.796638 392 external_mini_cluster.cc:1428] Setting key 412f3676f4486af0205a8cfeb2098c19
I20260924 05:08:51.799095 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:51.790813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58083 (local address 127.0.98.62:34049)
0924 05:08:51.790857 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:51.790859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:51.791089 (+ 230us) server_negotiation.cc:407] Connection header received
0924 05:08:51.791662 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:51.791663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:51.791676 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:51.791725 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:51.792674 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.793172 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.794156 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.794363 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.794444 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:51.796694 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:51.796696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:51.796696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:51.796704 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:51.798019 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.798346 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.798346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.798347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.798393 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.798581 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.798582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.798582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.798671 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:51.798727 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:51.798981 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 05:08:51.799035 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:08:51.799311 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:51.790830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34049 (local address 127.0.98.3:58083)
0924 05:08:51.790969 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:51.791004 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:51.791510 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:51.791796 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:51.791800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:51.792366 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:51.792566 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:51.792571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.793268 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.793270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:51.794066 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:51.794068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.794452 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.794453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:51.794581 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:51.795060 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:51.795068 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:51.796604 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:51.798085 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:51.798090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:51.798091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:51.798285 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:51.798443 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:51.798444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:51.798444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:51.798532 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:51.798727 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:51.798731 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:51.798883 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 05:08:51.798959 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 05:08:51.799075 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260924 05:08:51.799778 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34049
I20260924 05:08:51.799863 903 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:51.800028 903 heartbeater.cc:511] Master 127.0.98.62:34049 requested a full tablet report, sending...
I20260924 05:08:51.800323 444 ts_manager.cc:194] Registered new tserver with Master: 6270acc9a84548268f79c42fb68ae6e7 (127.0.98.3:40247)
I20260924 05:08:51.800631 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58083
I20260924 05:08:51.808275 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:08:51.830257 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:51.823380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.0.98.1:41959)
0924 05:08:51.823545 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:51.823549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:51.823560 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:08:51.823584 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:51.823586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:51.823599 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:51.823665 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:51.824170 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.824838 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.825542 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.825820 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.825870 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:51.827984 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:51.827991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:51.827991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:51.827997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:51.829273 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.829584 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.829586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.829586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.829638 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.829788 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.829789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.829789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.829899 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:51.829996 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:51.830072 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 05:08:51.830130 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
W20260924 05:08:51.830631 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47700
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:08:51.852016 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:51.845498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.0.98.1:41959)
0924 05:08:51.845557 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:51.845559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:51.845638 (+ 79us) server_negotiation.cc:407] Connection header received
0924 05:08:51.845754 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:51.845755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:51.845767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:51.845828 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:51.846370 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.846900 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.847642 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:51.847857 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:51.847954 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:51.850012 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:51.850013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:51.850013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:51.850019 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:51.851226 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.851509 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.851510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.851510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.851554 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:51.851687 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:51.851687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:51.851688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:51.851762 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:51.851840 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:51.851900 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 05:08:51.851954 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:08:51.852519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 506
I20260924 05:08:51.856948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 640
I20260924 05:08:51.862640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 774
I20260924 05:08:51.866864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 412
2026-09-24T05:08:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4133 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[917](info): set up 2 sockets
Sep 24 05:08:51 dist-test-slave-mr2f krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 05:08:52.878234 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:52 dist-test-slave-mr2f krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:52.989182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:53.079367 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:53.079543 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:53.079564 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:53.080559 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:53.080588 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:53.080601 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:53.080612 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:53.080623 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:53.082010 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:53.082185 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:53.082338 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:53.087107 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:53.087201 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:53.087205 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:53.087379 930 server_base.cc:1034] running on GCE node
I20260924 05:08:53.087625 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:53.087905 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:53.089059 930 hybrid_clock.cc:648] HybridClock initialized: now 1790226533089039 us; error 32 us; skew 500 ppm
Sep 24 05:08:53 dist-test-slave-mr2f krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:53.091445 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:53.091822 930 webserver.cc:460] Webserver started at http://127.0.98.62:39851/ using document root <none> and password file <none>
I20260924 05:08:53.092027 930 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:53.092097 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:53.092233 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:53.093221 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "4cde51325a2a4eb09ca9b956d49468e6"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "c80a4b93a88f87cc67e13e745c36121d"
server_key_iv: "5163c78f23443c433bff25f9bb9f86fa"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.093564 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "4cde51325a2a4eb09ca9b956d49468e6"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "c80a4b93a88f87cc67e13e745c36121d"
server_key_iv: "5163c78f23443c433bff25f9bb9f86fa"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.094872 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:08:53.095568 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.095758 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:53.095853 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "4cde51325a2a4eb09ca9b956d49468e6"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "c80a4b93a88f87cc67e13e745c36121d"
server_key_iv: "5163c78f23443c433bff25f9bb9f86fa"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.095932 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:53.102509 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:53.104244 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:53.104384 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:53.108112 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42067
I20260924 05:08:53.108155 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42067 every 8 connection(s)
I20260924 05:08:53.108477 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:53.109290 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:53.111368 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6: Bootstrap starting.
I20260924 05:08:53.111848 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:53.112092 998 log.cc:824] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:53.112668 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6: No bootstrap required, opened a new log
I20260924 05:08:53.114058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 930
I20260924 05:08:53.114169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:53.114149 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cde51325a2a4eb09ca9b956d49468e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } }
I20260924 05:08:53.114351 392 external_mini_cluster.cc:1428] Setting key e22061b982a5ade64dcb145e761c3837
I20260924 05:08:53.114336 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:53.114384 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cde51325a2a4eb09ca9b956d49468e6, State: Initialized, Role: FOLLOWER
I20260924 05:08:53.114547 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cde51325a2a4eb09ca9b956d49468e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } }
I20260924 05:08:53.114643 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:08:53.114681 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:08:53.114734 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:53.115597 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cde51325a2a4eb09ca9b956d49468e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } }
I20260924 05:08:53.115718 998 leader_election.cc:302] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cde51325a2a4eb09ca9b956d49468e6; no voters:
I20260924 05:08:53.115899 998 leader_election.cc:288] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:08:53.115929 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:53.116026 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [term 1 LEADER]: Becoming Leader. State: Replica: 4cde51325a2a4eb09ca9b956d49468e6, State: Running, Role: LEADER
I20260924 05:08:53.116107 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cde51325a2a4eb09ca9b956d49468e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } }
I20260924 05:08:53.116174 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:53.116437 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cde51325a2a4eb09ca9b956d49468e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cde51325a2a4eb09ca9b956d49468e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } }
I20260924 05:08:53.116504 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:53.116621 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cde51325a2a4eb09ca9b956d49468e6. Latest consensus state: current_term: 1 leader_uuid: "4cde51325a2a4eb09ca9b956d49468e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cde51325a2a4eb09ca9b956d49468e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42067 } } }
I20260924 05:08:53.116688 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:53.116966 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:53.117471 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:53.119050 1007 catalog_manager.cc:1347] Generated new cluster ID: c285c8ac380f46618e5bb0ab48c7c079
I20260924 05:08:53.119100 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:08:53 dist-test-slave-mr2f krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:53.125522 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.114822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.0.98.62:42067)
0924 05:08:53.115031 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:53.115041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:53.115097 (+ 56us) server_negotiation.cc:407] Connection header received
0924 05:08:53.115205 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:53.115210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:53.115250 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:53.115383 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:53.116096 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.117465 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.118540 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.119164 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.119290 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.121952 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:53.121957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:53.121957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:53.121968 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:53.123901 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.124284 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.124287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.124288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.124415 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.124602 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.124603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.124604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.124930 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:53.125132 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:53.125200 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 05:08:53.125334 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260924 05:08:53.132686 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:53.133394 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:53.143102 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cde51325a2a4eb09ca9b956d49468e6: Generated new TSK 0
I20260924 05:08:53.143262 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:08:53.167659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:53.245091 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:53.245302 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:53.245342 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:53.246449 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:53.246510 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:53.246554 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:08:53.248044 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:53.248294 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:53.248521 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:53.253317 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:53.253331 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:53.253477 1024 server_base.cc:1034] running on GCE node
W20260924 05:08:53.253331 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:53.253762 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:53.254027 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:53.255168 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790226533255161 us; error 37 us; skew 500 ppm
Sep 24 05:08:53 dist-test-slave-mr2f krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:53.257583 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:08:53.258006 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36787/ using document root <none> and password file <none>
I20260924 05:08:53.258211 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:53.258272 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:53.258409 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:53.259466 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "a73388df83ec4feea0af182c807a2eaf"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "b35d2740d2fb71ea8e580e61a3211b9f"
server_key_iv: "f3ac98b48ab8f41b2ae3facd34d5694c"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.259807 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a73388df83ec4feea0af182c807a2eaf"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "b35d2740d2fb71ea8e580e61a3211b9f"
server_key_iv: "f3ac98b48ab8f41b2ae3facd34d5694c"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.261106 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:08:53.261833 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.261986 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:08:53.262077 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "a73388df83ec4feea0af182c807a2eaf"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "b35d2740d2fb71ea8e580e61a3211b9f"
server_key_iv: "f3ac98b48ab8f41b2ae3facd34d5694c"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.262154 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:53.282884 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:53.285243 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:53.285413 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:53.285640 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:08:53.285933 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:53.285992 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.286038 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:53.286063 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.291765 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38509
I20260924 05:08:53.291803 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38509 every 8 connection(s)
I20260924 05:08:53.292129 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:08:53 dist-test-slave-mr2f krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:53.300925 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.292756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55741 (local address 127.0.98.62:42067)
0924 05:08:53.292799 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:53.292800 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:53.293004 (+ 204us) server_negotiation.cc:407] Connection header received
0924 05:08:53.293467 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:53.293468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:53.293481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:53.293546 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:53.294535 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.295156 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.296090 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.296280 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.296329 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.298442 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:53.298443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:53.298443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:53.298451 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:53.299856 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.300205 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.300205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.300206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.300261 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.300427 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.300428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.300428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.300518 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:53.300566 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:53.300801 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 05:08:53.300863 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:08:53.301119 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.292723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.1:55741)
0924 05:08:53.292884 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:53.292917 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:53.293390 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:53.293608 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:53.293611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:53.294137 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:53.294331 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.294335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.295246 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.295247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.296022 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.296024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.296344 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.296345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.296438 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.296864 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:53.296872 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:53.298366 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:53.299921 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:53.299927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:53.299928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:53.300140 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:53.300303 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:53.300303 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:53.300304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:53.300385 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:53.300566 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:53.300570 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:53.300690 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 05:08:53.300778 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 05:08:53.300892 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260924 05:08:53.301554 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42067
I20260924 05:08:53.301643 1153 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:53.301783 1153 heartbeater.cc:511] Master 127.0.98.62:42067 requested a full tablet report, sending...
I20260924 05:08:53.302088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 1024
I20260924 05:08:53.302207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:08:53.302307 962 ts_manager.cc:194] Registered new tserver with Master: a73388df83ec4feea0af182c807a2eaf (127.0.98.1:38509)
I20260924 05:08:53.302407 392 external_mini_cluster.cc:1428] Setting key 99770d6af8d15bc0a472244b890b31b5
I20260924 05:08:53.303076 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55741
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:08:53.325176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:53.403445 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:53.403631 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:53.403653 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:53.404786 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:53.404827 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:53.404892 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:08:53.406294 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:53.406520 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:53.406731 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:53.411394 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:53.411533 1158 server_base.cc:1034] running on GCE node
W20260924 05:08:53.411401 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:53.411412 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:53.411850 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:53.412079 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:53.413228 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790226533413223 us; error 34 us; skew 500 ppm
Sep 24 05:08:53 dist-test-slave-mr2f krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:53.415465 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:08:53.415901 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35539/ using document root <none> and password file <none>
I20260924 05:08:53.416105 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:53.416165 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:53.416319 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:53.417306 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "acb2f1353ce046f6b300b061f1c17916"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "a6c4d31819805ea783db3702c911a11b"
server_key_iv: "f28720f6285293424f388468ca0da62c"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.417634 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "acb2f1353ce046f6b300b061f1c17916"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "a6c4d31819805ea783db3702c911a11b"
server_key_iv: "f28720f6285293424f388468ca0da62c"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.418948 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:08:53.419633 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.419790 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.419886 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "acb2f1353ce046f6b300b061f1c17916"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "a6c4d31819805ea783db3702c911a11b"
server_key_iv: "f28720f6285293424f388468ca0da62c"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.419958 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:53.429593 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:53.431227 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:53.431371 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:53.431562 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:08:53.431838 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:53.431886 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.431916 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:53.431942 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.437278 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37071
I20260924 05:08:53.437335 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37071 every 8 connection(s)
I20260924 05:08:53.437609 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:08:53.438982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 1158
I20260924 05:08:53.439072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:08:53.439213 392 external_mini_cluster.cc:1428] Setting key 8ceef93233aa748da9f11d28e33b8b31
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 24 05:08:53 dist-test-slave-mr2f krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:53.452783 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.438204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.2:40163)
0924 05:08:53.438366 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:53.438396 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:53.438856 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:53.439091 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:53.439095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:53.439827 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:53.440032 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.440036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.440686 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.440687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.441954 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.441956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.442305 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.442306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.442445 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.448418 (+ 5973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:53.448428 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:53.449798 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:53.451485 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:53.451490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:53.451492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:53.451762 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:53.451971 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:53.451972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:53.451972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:53.452060 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:53.452350 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:53.452353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:53.452455 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 05:08:53.452527 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:08:53.452636 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260924 05:08:53.452795 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.438211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40163 (local address 127.0.98.62:42067)
0924 05:08:53.438279 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:53.438280 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:53.438470 (+ 190us) server_negotiation.cc:407] Connection header received
0924 05:08:53.438954 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:53.438956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:53.438968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:53.439028 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:53.440118 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.440601 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.442048 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.442231 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.442283 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.449891 (+ 7608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:53.449893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:53.449893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:53.449902 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:53.451408 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.451837 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.451838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.451839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.451903 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.452121 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.452122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.452122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.452279 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:53.452352 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:53.452572 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 05:08:53.452664 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:08:53.453272 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42067
I20260924 05:08:53.453356 1287 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:53.453531 1287 heartbeater.cc:511] Master 127.0.98.62:42067 requested a full tablet report, sending...
I20260924 05:08:53.453890 962 ts_manager.cc:194] Registered new tserver with Master: acb2f1353ce046f6b300b061f1c17916 (127.0.98.2:37071)
I20260924 05:08:53.454219 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:08:53.462865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:53.539813 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:53.540009 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:53.540043 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:53.541044 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:53.541100 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:53.541147 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:08:53.542498 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:53.542744 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:53.542968 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:53.547487 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:53.547495 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:53.547526 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:53.547638 1292 server_base.cc:1034] running on GCE node
I20260924 05:08:53.547834 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:53.548039 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:53.549170 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790226533549162 us; error 26 us; skew 500 ppm
Sep 24 05:08:53 dist-test-slave-mr2f krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:53.551342 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:08:53.551729 1292 webserver.cc:460] Webserver started at http://127.0.98.3:36939/ using document root <none> and password file <none>
I20260924 05:08:53.551906 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:53.551944 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:53.552023 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:53.552943 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "874f94f42091bb465e4c37d4d9898ca2"
server_key_iv: "571037230d53308815f1b395f9142cf9"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.553237 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "874f94f42091bb465e4c37d4d9898ca2"
server_key_iv: "571037230d53308815f1b395f9142cf9"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.554392 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:08:53.555019 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.555198 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.555276 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4"
format_stamp: "Formatted at 2026-09-24 05:08:53 on dist-test-slave-mr2f"
server_key: "874f94f42091bb465e4c37d4d9898ca2"
server_key_iv: "571037230d53308815f1b395f9142cf9"
server_key_version: "encryptionkey@0"
I20260924 05:08:53.555325 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:53.574893 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:53.576606 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:53.576707 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:53.576874 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:08:53.577088 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:53.577118 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.577138 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:53.577150 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:53.582485 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44605
I20260924 05:08:53.582554 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44605 every 8 connection(s)
I20260924 05:08:53.582852 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:08:53.587065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 1292
I20260924 05:08:53.587141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:08:53.587299 392 external_mini_cluster.cc:1428] Setting key ad65bede0abb916c74661dfef3a3a688
Sep 24 05:08:53 dist-test-slave-mr2f krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:53.591543 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.583513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41831 (local address 127.0.98.62:42067)
0924 05:08:53.583591 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:53.583594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:53.583753 (+ 159us) server_negotiation.cc:407] Connection header received
0924 05:08:53.584239 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:53.584240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:53.584253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:53.584300 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:53.585171 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.585668 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.586631 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.586896 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.586969 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.589184 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:53.589185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:53.589185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:53.589192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:53.590390 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.590722 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.590723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.590723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.590815 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.591011 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.591012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.591012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.591106 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:53.591156 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:53.591431 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 05:08:53.591484 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:08:53.591776 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.583479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42067 (local address 127.0.98.3:41831)
0924 05:08:53.583639 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:53.583675 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:53.584137 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:53.584379 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:53.584382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:53.584901 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:53.585071 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.585075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.585753 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.585754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.586541 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.586543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.586982 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.586983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.587090 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.587641 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:53.587651 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:53.589108 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:53.590453 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:53.590457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:53.590459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:53.590663 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:53.590872 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:53.590872 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:53.590872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:53.590960 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:53.591200 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:53.591206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:53.591316 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 05:08:53.591410 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 05:08:53.591520 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260924 05:08:53.592298 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42067
I20260924 05:08:53.592411 1421 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:53.592584 1421 heartbeater.cc:511] Master 127.0.98.62:42067 requested a full tablet report, sending...
I20260924 05:08:53.592941 962 ts_manager.cc:194] Registered new tserver with Master: e70a30b9b9cb4a71a33a236bcb6c3ec4 (127.0.98.3:44605)
I20260924 05:08:53.593318 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41831
I20260924 05:08:53.598830 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:08:53.605499 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.600390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.0.98.62:42067)
0924 05:08:53.600444 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:53.600446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:53.600546 (+ 100us) server_negotiation.cc:407] Connection header received
0924 05:08:53.600600 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:53.600600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:53.600608 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:53.600644 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:53.601093 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.601530 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.602160 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.602323 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.602383 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.603404 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:53.603405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:53.603406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:53.603411 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:53.604687 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.604940 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.604941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.604941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.604985 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:53.605135 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:53.605136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:53.605136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:53.605245 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:53.605311 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:53.605395 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 05:08:53.605445 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:08:53.607345 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:08:53.608798 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:08:53.614636 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.610610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44605 (local address 127.0.0.1:56366)
0924 05:08:53.610767 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:53.610821 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:53.610882 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:53.611575 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:53.611576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:53.611578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:53.611615 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.611617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.612480 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.612481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.613382 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.613385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.614349 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.614351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.614426 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.614442 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 05:08:53.614505 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:08:53.614555 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":49,"thread_start_us":61,"threads_started":1}
I20260924 05:08:53.614732 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.610934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.0.98.2:37071)
0924 05:08:53.611105 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:53.611109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:53.611124 (+ 15us) server_negotiation.cc:407] Connection header received
0924 05:08:53.611153 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:53.611154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:53.611165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:53.611245 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:53.611388 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.612427 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.613877 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.614394 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.614462 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.614514 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:53.614611 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 05:08:53.614645 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260924 05:08:53.614954 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.610664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.0.98.3:44605)
0924 05:08:53.611386 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:53.611390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:53.611398 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:08:53.611427 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:53.611429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:53.611441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:53.611520 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:53.611731 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.612382 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.613555 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.614265 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.614337 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.614388 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:53.614856 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 05:08:53.614893 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":77,"threads_started":1}
I20260924 05:08:53.614954 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.610619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38509 (local address 127.0.0.1:47506)
0924 05:08:53.610939 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:53.610950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:53.611000 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:53.611154 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:53.611157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:53.611159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:53.611213 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.611214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.612504 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.612505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.613780 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.613781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.614692 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.614693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.614799 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.614802 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:08:53.614845 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:08:53.614873 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":128,"thread_start_us":56,"threads_started":1}
I20260924 05:08:53.615181 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.610832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37071 (local address 127.0.0.1:42962)
0924 05:08:53.610987 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:53.610995 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:53.611032 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:53.611245 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:53.611246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:53.611248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:53.611295 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.611297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.612519 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.612520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.613776 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:53.613779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.614465 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.614466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:53.614523 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.614528 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:08:53.614598 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:08:53.614638 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":37,"thread_start_us":72,"threads_started":1}
I20260924 05:08:53.615271 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:53.610733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.98.1:38509)
0924 05:08:53.610997 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:53.611002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:53.611013 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:08:53.611056 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:53.611058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:53.611071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:53.611141 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:53.611303 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.612385 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.613860 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:53.614546 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:53.614614 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:53.614661 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:53.615137 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 05:08:53.615180 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":87,"threads_started":1}
I20260924 05:08:53.615568 1355 tablet_service.cc:1467] Processing CreateTablet for tablet fbd35e06ee044b129a989456eb24ce92 (DEFAULT_TABLE table=test-table [id=8b81006ed5d4429081c0234db52fc5ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:53.615933 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd35e06ee044b129a989456eb24ce92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:53.615850 1221 tablet_service.cc:1467] Processing CreateTablet for tablet fbd35e06ee044b129a989456eb24ce92 (DEFAULT_TABLE table=test-table [id=8b81006ed5d4429081c0234db52fc5ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:53.616149 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd35e06ee044b129a989456eb24ce92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:53.619072 1087 tablet_service.cc:1467] Processing CreateTablet for tablet fbd35e06ee044b129a989456eb24ce92 (DEFAULT_TABLE table=test-table [id=8b81006ed5d4429081c0234db52fc5ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:53.619421 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd35e06ee044b129a989456eb24ce92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:53.619410 1436 tablet_bootstrap.cc:492] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4: Bootstrap starting.
I20260924 05:08:53.619983 1436 tablet_bootstrap.cc:654] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:53.620512 1436 log.cc:824] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:53.620807 1435 tablet_bootstrap.cc:492] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916: Bootstrap starting.
I20260924 05:08:53.621141 1436 tablet_bootstrap.cc:492] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4: No bootstrap required, opened a new log
I20260924 05:08:53.621212 1436 ts_tablet_manager.cc:1397] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:08:53.621361 1435 tablet_bootstrap.cc:654] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:53.621855 1435 log.cc:824] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:53.622609 1435 tablet_bootstrap.cc:492] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916: No bootstrap required, opened a new log
I20260924 05:08:53.622597 1439 tablet_bootstrap.cc:492] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf: Bootstrap starting.
I20260924 05:08:53.622676 1435 ts_tablet_manager.cc:1397] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:08:53.622846 1436 raft_consensus.cc:348] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } }
I20260924 05:08:53.623472 1436 raft_consensus.cc:374] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:53.623536 1436 raft_consensus.cc:729] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70a30b9b9cb4a71a33a236bcb6c3ec4, State: Initialized, Role: FOLLOWER
I20260924 05:08:53.623674 1436 consensus_queue.cc:260] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } }
I20260924 05:08:53.623848 1439 tablet_bootstrap.cc:654] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:53.624004 1436 ts_tablet_manager.cc:1428] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:08:53.624169 1421 heartbeater.cc:503] Master 127.0.98.62:42067 was elected leader, sending a full tablet report...
I20260924 05:08:53.624393 1439 log.cc:824] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:53.624936 1435 raft_consensus.cc:348] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } }
I20260924 05:08:53.625100 1439 tablet_bootstrap.cc:492] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf: No bootstrap required, opened a new log
I20260924 05:08:53.625141 1439 ts_tablet_manager.cc:1397] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:08:53.625118 1435 raft_consensus.cc:374] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:53.625168 1435 raft_consensus.cc:729] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb2f1353ce046f6b300b061f1c17916, State: Initialized, Role: FOLLOWER
I20260924 05:08:53.625285 1435 consensus_queue.cc:260] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } }
I20260924 05:08:53.625550 1435 ts_tablet_manager.cc:1428] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:08:53.625578 1287 heartbeater.cc:503] Master 127.0.98.62:42067 was elected leader, sending a full tablet report...
I20260924 05:08:53.626971 1439 raft_consensus.cc:348] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } }
I20260924 05:08:53.627089 1439 raft_consensus.cc:374] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:53.627110 1439 raft_consensus.cc:729] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a73388df83ec4feea0af182c807a2eaf, State: Initialized, Role: FOLLOWER
I20260924 05:08:53.627177 1439 consensus_queue.cc:260] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } }
I20260924 05:08:53.627377 1439 ts_tablet_manager.cc:1428] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:08:53.627410 1153 heartbeater.cc:503] Master 127.0.98.62:42067 was elected leader, sending a full tablet report...
W20260924 05:08:53.688252 1288 tablet.cc:2369] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:08:53.792778 1154 tablet.cc:2369] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:08:53.833415 1422 tablet.cc:2369] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:08:54.022311 1440 raft_consensus.cc:482] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:08:54.022421 1440 raft_consensus.cc:504] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } }
I20260924 05:08:54.022753 1440 leader_election.cc:288] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a73388df83ec4feea0af182c807a2eaf (127.0.98.1:38509), acb2f1353ce046f6b300b061f1c17916 (127.0.98.2:37071)
I20260924 05:08:54.025803 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:54.023077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34961 (local address 127.0.98.1:38509)
0924 05:08:54.023135 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:54.023137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:54.023149 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:08:54.023281 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:54.023282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:54.023295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:54.023348 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:54.023540 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.024115 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.025070 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.025477 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.025530 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:54.025539 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:54.025689 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 05:08:54.025729 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:08:54.025811 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:54.023006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38509 (local address 127.0.98.3:34961)
0924 05:08:54.023060 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:54.023073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:54.023173 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:54.023362 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:54.023364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:54.023365 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:54.023440 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:54.023442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.024185 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.024186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:54.024984 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:54.024986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.025545 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.025546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:54.025622 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:54.025626 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:08:54.025677 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:08:54.025715 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:08:54.025885 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:54.023284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33981 (local address 127.0.98.2:37071)
0924 05:08:54.023343 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:54.023345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:54.023422 (+ 77us) server_negotiation.cc:407] Connection header received
0924 05:08:54.023568 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:54.023568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:54.023578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:54.023619 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:54.023763 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.024232 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.025173 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.025545 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.025611 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:54.025620 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:54.025779 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 05:08:54.025814 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:08:54.025936 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:54.023191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37071 (local address 127.0.98.3:33981)
0924 05:08:54.023351 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:54.023380 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:54.023428 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:54.023627 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:54.023627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:54.023630 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:54.023692 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:54.023700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.024323 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.024323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:54.025086 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:54.025087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.025629 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.025629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:54.025704 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:54.025708 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:08:54.025761 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 05:08:54.025796 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260924 05:08:54.026177 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd35e06ee044b129a989456eb24ce92" candidate_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a73388df83ec4feea0af182c807a2eaf" is_pre_election: true
I20260924 05:08:54.026207 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd35e06ee044b129a989456eb24ce92" candidate_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb2f1353ce046f6b300b061f1c17916" is_pre_election: true
I20260924 05:08:54.026330 1241 raft_consensus.cc:2454] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e70a30b9b9cb4a71a33a236bcb6c3ec4 in term 0.
I20260924 05:08:54.026330 1107 raft_consensus.cc:2454] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e70a30b9b9cb4a71a33a236bcb6c3ec4 in term 0.
I20260924 05:08:54.026523 1311 leader_election.cc:302] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a73388df83ec4feea0af182c807a2eaf, e70a30b9b9cb4a71a33a236bcb6c3ec4; no voters:
I20260924 05:08:54.026644 1440 raft_consensus.cc:2781] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:08:54.026695 1440 raft_consensus.cc:482] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:08:54.026718 1440 raft_consensus.cc:3037] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:54.027442 1440 raft_consensus.cc:504] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } }
I20260924 05:08:54.027566 1440 leader_election.cc:288] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [CANDIDATE]: Term 1 election: Requested vote from peers a73388df83ec4feea0af182c807a2eaf (127.0.98.1:38509), acb2f1353ce046f6b300b061f1c17916 (127.0.98.2:37071)
I20260924 05:08:54.027725 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd35e06ee044b129a989456eb24ce92" candidate_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a73388df83ec4feea0af182c807a2eaf"
I20260924 05:08:54.027737 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd35e06ee044b129a989456eb24ce92" candidate_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acb2f1353ce046f6b300b061f1c17916"
I20260924 05:08:54.027804 1241 raft_consensus.cc:3037] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:54.027804 1107 raft_consensus.cc:3037] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:54.028631 1241 raft_consensus.cc:2454] T fbd35e06ee044b129a989456eb24ce92 P acb2f1353ce046f6b300b061f1c17916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e70a30b9b9cb4a71a33a236bcb6c3ec4 in term 1.
I20260924 05:08:54.028631 1107 raft_consensus.cc:2454] T fbd35e06ee044b129a989456eb24ce92 P a73388df83ec4feea0af182c807a2eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e70a30b9b9cb4a71a33a236bcb6c3ec4 in term 1.
I20260924 05:08:54.028784 1311 leader_election.cc:302] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acb2f1353ce046f6b300b061f1c17916, e70a30b9b9cb4a71a33a236bcb6c3ec4; no voters:
I20260924 05:08:54.028869 1440 raft_consensus.cc:2781] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:54.028999 1440 raft_consensus.cc:686] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [term 1 LEADER]: Becoming Leader. State: Replica: e70a30b9b9cb4a71a33a236bcb6c3ec4, State: Running, Role: LEADER
I20260924 05:08:54.029090 1440 consensus_queue.cc:237] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } }
I20260924 05:08:54.029954 961 catalog_manager.cc:5495] T fbd35e06ee044b129a989456eb24ce92 P e70a30b9b9cb4a71a33a236bcb6c3ec4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e70a30b9b9cb4a71a33a236bcb6c3ec4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70a30b9b9cb4a71a33a236bcb6c3ec4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a73388df83ec4feea0af182c807a2eaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acb2f1353ce046f6b300b061f1c17916" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37071 } health_report { overall_health: UNKNOWN } } }
Sep 24 05:08:54 dist-test-slave-mr2f krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:08:54.055761 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:54.048993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.0.98.1:38509)
0924 05:08:54.049057 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:54.049059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:54.049145 (+ 86us) server_negotiation.cc:407] Connection header received
0924 05:08:54.049214 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:54.049215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:54.049224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:54.049284 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:54.049759 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.050165 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.050893 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.051060 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.051132 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:54.053237 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:54.053246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:54.053246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:54.053253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:54.054604 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:54.054990 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:54.054993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:54.054993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:54.055060 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:54.055267 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:54.055267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:54.055268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:54.055394 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:54.055539 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:54.055595 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 05:08:54.055664 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 05:08:54 dist-test-slave-mr2f krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:08:54 dist-test-slave-mr2f krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:08:54.077117 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:54.071068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.0.98.1:38509)
0924 05:08:54.071137 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:54.071140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:54.071196 (+ 56us) server_negotiation.cc:407] Connection header received
0924 05:08:54.071237 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:54.071237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:54.071245 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:54.071291 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:54.071744 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.072167 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.072814 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.072981 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.073028 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:54.075122 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:54.075123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:54.075123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:54.075129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:54.076212 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:54.076504 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:54.076505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:54.076505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:54.076548 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:54.076748 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:54.076748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:54.076748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:54.076832 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:54.076902 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:54.077009 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 05:08:54.077060 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 05:08:54 dist-test-slave-mr2f krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:08:54 dist-test-slave-mr2f krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:08:54.097295 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:54.091207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.98.1:38509)
0924 05:08:54.091290 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:54.091292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:54.091352 (+ 60us) server_negotiation.cc:407] Connection header received
0924 05:08:54.091403 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:54.091404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:54.091411 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:54.091457 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:54.091913 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.092331 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.092992 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:54.093160 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:54.093212 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:54.095294 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:54.095295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:54.095295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:54.095301 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:54.096401 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:54.096701 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:54.096702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:54.096702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:54.096745 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:54.096883 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:54.096883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:54.096883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:54.096960 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:54.097042 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:54.097154 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 05:08:54.097207 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 05:08:54.097724 1067 tablet_service.cc:2908] Rejecting scan request for tablet fbd35e06ee044b129a989456eb24ce92: Uninitialized: clean time has not yet been initialized
I20260924 05:08:54.098817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 1024
I20260924 05:08:54.103739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 1158
I20260924 05:08:54.107919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 1292
I20260924 05:08:54.112638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 930
2026-09-24T05:08:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2247 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:54 dist-test-slave-mr2f krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:54 dist-test-slave-mr2f krb5kdc[1459](info): set up 2 sockets
Sep 24 05:08:54 dist-test-slave-mr2f krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260924 05:08:55.126405 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:55.242185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38327
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:55.320880 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:55.321099 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:55.321135 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:55.322305 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:55.322367 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 05:08:55.322393 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:55.322425 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:55.322450 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:55.322480 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:55.323980 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38327
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:55.324254 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:55.324462 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:08:55.328951 1472 server_base.cc:1034] running on GCE node
W20260924 05:08:55.328902 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:55.328903 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:55.328918 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:55.329397 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:55.329685 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:55.330821 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790226535330802 us; error 29 us; skew 500 ppm
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:55.333308 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:55.333730 1472 webserver.cc:460] Webserver started at http://127.0.98.62:44799/ using document root <none> and password file <none>
I20260924 05:08:55.333941 1472 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:55.334007 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:55.334151 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:55.335263 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "c2aa0952bf604aeaad9a0f480b0004c1"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "a9c978fa4a6723f497d7611ac41dd514"
server_key_iv: "da33710218241756d1d5290805c68bad"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.335611 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "c2aa0952bf604aeaad9a0f480b0004c1"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "a9c978fa4a6723f497d7611ac41dd514"
server_key_iv: "da33710218241756d1d5290805c68bad"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.336992 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:08:55.337643 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.337800 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:55.337895 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "c2aa0952bf604aeaad9a0f480b0004c1"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "a9c978fa4a6723f497d7611ac41dd514"
server_key_iv: "da33710218241756d1d5290805c68bad"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.337970 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:55.345932 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:55.348080 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:55.348230 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:55.352609 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38327
I20260924 05:08:55.352659 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38327 every 8 connection(s)
I20260924 05:08:55.352942 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:55.353511 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:55.355695 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1: Bootstrap starting.
I20260924 05:08:55.356292 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:55.356379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 1472
I20260924 05:08:55.356498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:55.356664 392 external_mini_cluster.cc:1428] Setting key 83e352d0604d09debdfd4b30ee37ff3e
I20260924 05:08:55.356649 1550 log.cc:824] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:55.357352 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1: No bootstrap required, opened a new log
I20260924 05:08:55.359325 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2aa0952bf604aeaad9a0f480b0004c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } }
I20260924 05:08:55.359450 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:55.359470 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2aa0952bf604aeaad9a0f480b0004c1, State: Initialized, Role: FOLLOWER
I20260924 05:08:55.359624 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2aa0952bf604aeaad9a0f480b0004c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } }
I20260924 05:08:55.359702 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:08:55.359731 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:08:55.359774 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:55.360411 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2aa0952bf604aeaad9a0f480b0004c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } }
I20260924 05:08:55.360523 1550 leader_election.cc:302] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2aa0952bf604aeaad9a0f480b0004c1; no voters:
I20260924 05:08:55.360688 1550 leader_election.cc:288] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:08:55.360714 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:55.360816 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [term 1 LEADER]: Becoming Leader. State: Replica: c2aa0952bf604aeaad9a0f480b0004c1, State: Running, Role: LEADER
I20260924 05:08:55.360934 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:55.360968 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2aa0952bf604aeaad9a0f480b0004c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } }
I20260924 05:08:55.361332 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2aa0952bf604aeaad9a0f480b0004c1. Latest consensus state: current_term: 1 leader_uuid: "c2aa0952bf604aeaad9a0f480b0004c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2aa0952bf604aeaad9a0f480b0004c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } } }
I20260924 05:08:55.361459 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2aa0952bf604aeaad9a0f480b0004c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2aa0952bf604aeaad9a0f480b0004c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } } }
I20260924 05:08:55.361514 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:55.361905 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:55.361634 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:55.362949 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:55.364427 1564 catalog_manager.cc:1347] Generated new cluster ID: 2aaddfed53b840039a1cc8446a444c11
I20260924 05:08:55.364475 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:08:55.367056 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.357119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.0.98.62:38327)
0924 05:08:55.357268 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.357277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.357306 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:08:55.357758 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.357764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.357806 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.357938 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:55.358530 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.359418 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.360529 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.361111 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.361232 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.363933 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:55.363936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:55.363937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:55.363948 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:55.365577 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.365974 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.365977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.365978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.366063 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.366275 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.366275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.366276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.366502 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:55.366661 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.366736 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.366905 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260924 05:08:55.406803 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:55.407536 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:55.413949 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1: Generated new TSK 0
I20260924 05:08:55.414129 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:08:55.441905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38327
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:55.519763 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:55.519968 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:55.519992 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:55.521059 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:55.521093 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:55.521114 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:08:55.521131 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:08:55.522545 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:55.522724 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:55.522953 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:55.527720 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:55.527693 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:55.527698 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:55.527974 1576 server_base.cc:1034] running on GCE node
I20260924 05:08:55.528167 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:55.528411 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:55.529577 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790226535529558 us; error 31 us; skew 500 ppm
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:55.531962 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:08:55.532394 1576 webserver.cc:460] Webserver started at http://127.0.98.1:39089/ using document root <none> and password file <none>
I20260924 05:08:55.532631 1576 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:55.532704 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:55.532873 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:55.533861 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "8fcc2875ea3c42bd882631710ccbf2e0"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "08fe024a22a7dcb513305dc83aa49329"
server_key_iv: "31779e377e52b1801e59a3156b77ee19"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.534214 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8fcc2875ea3c42bd882631710ccbf2e0"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "08fe024a22a7dcb513305dc83aa49329"
server_key_iv: "31779e377e52b1801e59a3156b77ee19"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.535560 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:08:55.536275 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.536435 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.536531 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "8fcc2875ea3c42bd882631710ccbf2e0"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "08fe024a22a7dcb513305dc83aa49329"
server_key_iv: "31779e377e52b1801e59a3156b77ee19"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.536609 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:55.557530 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:55.559314 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:55.559473 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:55.559953 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:55.560014 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.560056 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:55.560086 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.566627 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40477
I20260924 05:08:55.566710 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40477 every 8 connection(s)
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:55.567049 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:08:55.569938 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.560501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55315 (local address 127.0.98.62:38327)
0924 05:08:55.560558 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.560560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.560728 (+ 168us) server_negotiation.cc:407] Connection header received
0924 05:08:55.561332 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.561333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.561354 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.561412 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:55.562233 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.563121 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.564642 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.564999 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.565089 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.567465 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:55.567466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:55.567466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:55.567474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:55.568856 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.569209 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.569210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.569211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.569258 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.569421 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.569422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.569422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.569508 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:55.569551 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.569802 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.569871 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:08:55.570068 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.560478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.98.1:55315)
0924 05:08:55.560618 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:55.560655 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:55.561231 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:55.561480 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:55.561484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:55.561869 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:55.562136 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.562142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.563213 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.563214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.564523 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.564526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.565092 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.565093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.565190 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.565660 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:55.565669 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:55.567358 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:55.568934 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.568938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.568940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.569143 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.569297 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.569298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.569298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.569382 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.569573 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:55.569578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:55.569688 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 05:08:55.569775 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 05:08:55.569885 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260924 05:08:55.570590 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38327
I20260924 05:08:55.570683 1707 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:55.570876 1707 heartbeater.cc:511] Master 127.0.98.62:38327 requested a full tablet report, sending...
I20260924 05:08:55.571368 1503 ts_manager.cc:194] Registered new tserver with Master: 8fcc2875ea3c42bd882631710ccbf2e0 (127.0.98.1:40477)
I20260924 05:08:55.572252 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55315
I20260924 05:08:55.576169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 1576
I20260924 05:08:55.576290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:08:55.576462 392 external_mini_cluster.cc:1428] Setting key 22d42860088df69f391a77e2108eb903
I20260924 05:08:55.576778 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.571554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54839 (local address 127.0.98.62:38327)
0924 05:08:55.571605 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.571607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.571727 (+ 120us) server_negotiation.cc:407] Connection header received
0924 05:08:55.571785 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.571786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.571792 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.571831 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:55.572141 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.572646 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.573355 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.573551 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.573632 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.574490 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:55.574491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:55.574492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:55.574498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:55.575674 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.576047 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.576048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.576048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.576101 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.576299 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.576300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.576300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.576426 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:55.576501 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.576634 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.576697 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:08:55.576795 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.571550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.98.1:54839)
0924 05:08:55.571663 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:55.571675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:55.571734 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:55.571842 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:55.571843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:55.572026 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:55.572087 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.572088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.572741 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.572742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.573266 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.573267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.573634 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.573635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.573702 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.574100 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:55.574105 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:55.574389 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:55.575771 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.575772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.575772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.575941 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.576169 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.576170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.576170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.576227 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.576494 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:55.576495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:55.576538 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 05:08:55.576599 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 05:08:55.576694 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:08:55.599479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38327
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:55.678406 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:55.678593 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:55.678615 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:55.679780 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:55.679833 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:55.679853 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:08:55.679889 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:08:55.681329 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:55.681528 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:55.681725 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:55.686499 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:55.686535 1716 server_base.cc:1034] running on GCE node
W20260924 05:08:55.686578 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:55.686508 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:55.686920 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:55.687165 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:55.688319 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790226535688318 us; error 34 us; skew 500 ppm
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:55.690835 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:08:55.691252 1716 webserver.cc:460] Webserver started at http://127.0.98.2:39673/ using document root <none> and password file <none>
I20260924 05:08:55.691478 1716 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:55.691541 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:55.691665 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:55.692658 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "6252d06350074e53a1d1998165d70bdb"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "17464f5feb866c8d70db9e4e971de316"
server_key_iv: "fa08df72c73fb4dceb71a59ffeaa33b3"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.693022 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6252d06350074e53a1d1998165d70bdb"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "17464f5feb866c8d70db9e4e971de316"
server_key_iv: "fa08df72c73fb4dceb71a59ffeaa33b3"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.694304 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:08:55.695111 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.695312 1716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:08:55.695410 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "6252d06350074e53a1d1998165d70bdb"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "17464f5feb866c8d70db9e4e971de316"
server_key_iv: "fa08df72c73fb4dceb71a59ffeaa33b3"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.695487 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:55.707275 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:55.709388 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:55.709533 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:55.709993 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:55.710050 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.710093 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:55.710125 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:55.717032 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43969
I20260924 05:08:55.717078 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43969 every 8 connection(s)
I20260924 05:08:55.717383 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:08:55.720073 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.710511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50223 (local address 127.0.98.62:38327)
0924 05:08:55.710567 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.710569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.710792 (+ 223us) server_negotiation.cc:407] Connection header received
0924 05:08:55.711305 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.711307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.711320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.711370 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:55.712013 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.712868 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.714520 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.714855 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.714926 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.717522 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:55.717523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:55.717524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:55.717532 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:55.718703 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.719129 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.719130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.719131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.719196 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.719450 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.719451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.719452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.719567 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:55.719642 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.719906 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.719980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:08:55.720257 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.710494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.98.2:50223)
0924 05:08:55.710639 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:55.710677 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:55.711192 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:55.711434 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:55.711437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:55.711745 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:55.711920 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.711924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.712970 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.712971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.714420 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.714423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.714983 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.714984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.715165 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.715644 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:55.715654 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:55.717418 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:55.718817 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.718822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.718824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.719038 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.719263 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.719264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.719264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.719382 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.719634 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:55.719639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:55.719779 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 05:08:55.719886 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 05:08:55.720037 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260924 05:08:55.720774 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38327
I20260924 05:08:55.720846 1847 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:55.721029 1847 heartbeater.cc:511] Master 127.0.98.62:38327 requested a full tablet report, sending...
I20260924 05:08:55.721413 1504 ts_manager.cc:194] Registered new tserver with Master: 6252d06350074e53a1d1998165d70bdb (127.0.98.2:43969)
I20260924 05:08:55.721797 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50223
I20260924 05:08:55.723974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 1716
I20260924 05:08:55.724062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:08:55.724268 392 external_mini_cluster.cc:1428] Setting key 3d6c6575c1ac46a75af1b464bd37c93c
I20260924 05:08:55.727685 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.722096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48587 (local address 127.0.98.62:38327)
0924 05:08:55.722131 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.722133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.722286 (+ 153us) server_negotiation.cc:407] Connection header received
0924 05:08:55.722309 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.722310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.722320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.722375 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:55.722640 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.723147 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.723839 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.724128 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.724229 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.725088 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:55.725089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:55.725089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:55.725096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:55.726628 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.727036 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.727037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.727037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.727078 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.727244 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.727244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.727244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.727353 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:55.727398 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.727578 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.727630 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:08:55.727828 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.722037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.98.2:48587)
0924 05:08:55.722191 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:55.722200 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:55.722249 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:55.722378 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:55.722378 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:55.722522 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:55.722574 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.722575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.723226 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.723227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.723736 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.723737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.724231 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.724232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.724306 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.724704 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:55.724708 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:55.725016 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:55.726707 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.726708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.726709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.726963 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.727119 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.727120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.727121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.727185 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.727434 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:55.727435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:55.727481 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:08:55.727560 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 05:08:55.727668 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:08:55.749043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38327
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:55.828238 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:55.828471 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:55.828510 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:55.829645 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:55.829713 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:55.829748 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:08:55.829788 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:08:55.831274 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:55.831525 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:55.831738 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:08:55.836676 1856 server_base.cc:1034] running on GCE node
W20260924 05:08:55.836583 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:55.836607 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:55.836580 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:55.836994 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:55.837271 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:55.838423 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790226535838409 us; error 27 us; skew 500 ppm
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:55.840996 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:08:55.841454 1856 webserver.cc:460] Webserver started at http://127.0.98.3:40263/ using document root <none> and password file <none>
I20260924 05:08:55.841666 1856 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:55.841711 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:55.841851 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:55.842985 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "62fa95f0b9984510b91f3e2c062e2bcb"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "9d635075c49683ff5f6e321bf035b3d3"
server_key_iv: "208be6877dcf4db61c8251e68bbf4070"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.843369 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "62fa95f0b9984510b91f3e2c062e2bcb"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "9d635075c49683ff5f6e321bf035b3d3"
server_key_iv: "208be6877dcf4db61c8251e68bbf4070"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.844645 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:08:55.845351 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.845530 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:55.845615 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "62fa95f0b9984510b91f3e2c062e2bcb"
format_stamp: "Formatted at 2026-09-24 05:08:55 on dist-test-slave-mr2f"
server_key: "9d635075c49683ff5f6e321bf035b3d3"
server_key_iv: "208be6877dcf4db61c8251e68bbf4070"
server_key_version: "encryptionkey@0"
I20260924 05:08:55.845682 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:55.862717 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:55.864743 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:55.864859 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:55.865348 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:55.865417 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.865449 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:55.865480 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:55.872671 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45095
I20260924 05:08:55.872756 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45095 every 8 connection(s)
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:55.873102 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:08:55.876266 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.865798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57581 (local address 127.0.98.62:38327)
0924 05:08:55.865867 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.865870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.866065 (+ 195us) server_negotiation.cc:407] Connection header received
0924 05:08:55.866559 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.866560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.866572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.866665 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:55.867609 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.868418 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.870232 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.870557 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.870664 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.873378 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:55.873380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:55.873380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:55.873388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:55.874890 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.875315 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.875317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.875317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.875385 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.875632 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.875633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.875634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.875751 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:55.875823 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.876099 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.876174 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:08:55.876457 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.865785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.98.3:57581)
0924 05:08:55.865931 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:55.865972 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:55.866461 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:55.866726 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:55.866730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:55.867184 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:55.867516 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.867524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.868522 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.868523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.870105 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.870108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.870670 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.870671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.870863 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.871440 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:55.871451 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:55.873275 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:55.874986 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.874992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.874994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.875225 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.875451 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.875452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.875452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.875564 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.875826 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:55.875832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:55.875971 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 05:08:55.876079 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 05:08:55.876234 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260924 05:08:55.876907 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38327
I20260924 05:08:55.876991 1987 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:55.877183 1987 heartbeater.cc:511] Master 127.0.98.62:38327 requested a full tablet report, sending...
I20260924 05:08:55.877624 1503 ts_manager.cc:194] Registered new tserver with Master: 62fa95f0b9984510b91f3e2c062e2bcb (127.0.98.3:45095)
I20260924 05:08:55.878085 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57581
I20260924 05:08:55.882622 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.877964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43687 (local address 127.0.98.62:38327)
0924 05:08:55.878204 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.878206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.878215 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:08:55.878234 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.878235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.878245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.878295 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:55.878637 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.879193 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.879836 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.880005 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.880053 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.880783 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:55.880784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:55.880784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:55.880789 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:55.881868 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.882138 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.882139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.882139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.882182 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.882323 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.882324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.882324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.882401 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:55.882442 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.882520 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.882568 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260924 05:08:55.882635 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.877900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.98.3:43687)
0924 05:08:55.878009 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:55.878019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:55.878068 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:55.878314 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:55.878315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:55.878503 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:55.878559 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.878561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.879275 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.879275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.879758 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.879759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.880063 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.880064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.880105 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.880460 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:55.880463 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:55.880712 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:55.881932 (+ 1220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.881933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.881933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.882078 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.882242 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:55.882243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:55.882243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:55.882288 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:55.882434 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:55.882435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:55.882463 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 05:08:55.882505 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:08:55.882568 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
I20260924 05:08:55.883249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 1856
I20260924 05:08:55.883312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:08:55.883503 392 external_mini_cluster.cc:1428] Setting key b7497a5feebca9d575441831da1f99f9
I20260924 05:08:55.884311 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:08:55 dist-test-slave-mr2f krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:55.908847 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.902067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.0.98.62:38327)
0924 05:08:55.902131 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.902134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.902147 (+ 13us) server_negotiation.cc:407] Connection header received
0924 05:08:55.902174 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.902200 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.902210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.902319 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:55.903114 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.903934 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.904694 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.904868 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.904927 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.906973 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:55.906974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:55.906974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:55.906980 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:55.908048 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.908347 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.908347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.908347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.908392 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:55.908523 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:55.908523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:55.908523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:55.908603 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:55.908668 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.908739 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.908788 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:08:55.910267 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:08:55.911852 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:08:55.918015 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.914045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40477 (local address 127.0.0.1:34092)
0924 05:08:55.914415 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:55.914424 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:55.914483 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:55.914699 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:55.914700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:55.914702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:55.914752 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.914754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.915649 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.915650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.916924 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.916927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.917672 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.917673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.917761 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.917776 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 05:08:55.917877 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 05:08:55.917930 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":21,"thread_start_us":48,"threads_started":1}
I20260924 05:08:55.918025 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.914226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.0.98.1:40477)
0924 05:08:55.914404 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.914409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.914468 (+ 59us) server_negotiation.cc:407] Connection header received
0924 05:08:55.914546 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.914548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.914562 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.914634 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:55.914865 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.915545 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.917012 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.917543 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.917669 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.917760 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.917895 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.917934 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
I20260924 05:08:55.918165 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.913865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43969 (local address 127.0.0.1:55894)
0924 05:08:55.914138 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:55.914150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:55.914211 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:55.914726 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:55.914727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:55.914730 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:55.914811 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.914813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.916072 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.916073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.916959 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.916961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.917928 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.917929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.917996 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.918000 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:08:55.918030 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 05:08:55.918067 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":1,"thread_start_us":55,"threads_started":1}
I20260924 05:08:55.918541 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.914033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55894 (local address 127.0.98.2:43969)
0924 05:08:55.914430 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.914434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.914444 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:55.914594 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.914598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.914617 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.914891 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:55.914953 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.915989 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.917058 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.917823 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.918265 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.918358 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.918389 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.918436 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":80,"threads_started":1}
I20260924 05:08:55.918589 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.913859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45095 (local address 127.0.0.1:36208)
0924 05:08:55.914018 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:55.914055 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:55.914114 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:55.914483 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:55.914487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:55.914489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:55.914547 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.914549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.915923 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.915924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.917006 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:55.917008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.918005 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.918006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:55.918421 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.918426 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:08:55.918476 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:08:55.918517 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":54,"threads_started":1}
I20260924 05:08:55.918875 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:55.913944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.0.98.3:45095)
0924 05:08:55.914252 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:55.914258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:55.914269 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:08:55.914310 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:55.914312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:55.914329 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:55.914404 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:55.914628 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.915836 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.917089 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:55.917823 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:55.917902 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:55.917977 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:55.918675 (+ 698us) server_negotiation.cc:299] Negotiation successful
0924 05:08:55.918714 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":92,"threads_started":1}
I20260924 05:08:55.918752 1641 tablet_service.cc:1467] Processing CreateTablet for tablet a562880f19144b0cb2c9e3abf028a7ea (DEFAULT_TABLE table=test-table [id=c0b366b263aa4a84913d9e8a2b346255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:55.919147 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a562880f19144b0cb2c9e3abf028a7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:55.922122 2007 tablet_bootstrap.cc:492] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: Bootstrap starting.
I20260924 05:08:55.922307 1781 tablet_service.cc:1467] Processing CreateTablet for tablet a562880f19144b0cb2c9e3abf028a7ea (DEFAULT_TABLE table=test-table [id=c0b366b263aa4a84913d9e8a2b346255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:55.922305 1921 tablet_service.cc:1467] Processing CreateTablet for tablet a562880f19144b0cb2c9e3abf028a7ea (DEFAULT_TABLE table=test-table [id=c0b366b263aa4a84913d9e8a2b346255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:55.922655 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a562880f19144b0cb2c9e3abf028a7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:55.922655 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a562880f19144b0cb2c9e3abf028a7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:55.922925 2007 tablet_bootstrap.cc:654] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:55.923270 2007 log.cc:824] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:55.923949 2007 tablet_bootstrap.cc:492] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: No bootstrap required, opened a new log
I20260924 05:08:55.924029 2007 ts_tablet_manager.cc:1397] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:08:55.925439 2007 raft_consensus.cc:348] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:55.925587 2007 raft_consensus.cc:374] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:55.925631 2007 raft_consensus.cc:729] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcc2875ea3c42bd882631710ccbf2e0, State: Initialized, Role: FOLLOWER
I20260924 05:08:55.925768 2007 consensus_queue.cc:260] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:55.926038 2007 ts_tablet_manager.cc:1428] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:08:55.926175 1707 heartbeater.cc:503] Master 127.0.98.62:38327 was elected leader, sending a full tablet report...
I20260924 05:08:55.926553 2011 tablet_bootstrap.cc:492] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: Bootstrap starting.
I20260924 05:08:55.926661 2010 tablet_bootstrap.cc:492] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: Bootstrap starting.
I20260924 05:08:55.927212 2011 tablet_bootstrap.cc:654] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:55.927212 2010 tablet_bootstrap.cc:654] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:55.927541 2010 log.cc:824] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:55.927541 2011 log.cc:824] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:55.928210 2010 tablet_bootstrap.cc:492] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: No bootstrap required, opened a new log
I20260924 05:08:55.928258 2011 tablet_bootstrap.cc:492] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: No bootstrap required, opened a new log
I20260924 05:08:55.928295 2010 ts_tablet_manager.cc:1397] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:08:55.928328 2011 ts_tablet_manager.cc:1397] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:08:55.929780 2010 raft_consensus.cc:348] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:55.929926 2010 raft_consensus.cc:374] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:55.929972 2010 raft_consensus.cc:729] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fa95f0b9984510b91f3e2c062e2bcb, State: Initialized, Role: FOLLOWER
I20260924 05:08:55.929893 2011 raft_consensus.cc:348] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:55.930052 2011 raft_consensus.cc:374] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:55.930097 2011 raft_consensus.cc:729] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6252d06350074e53a1d1998165d70bdb, State: Initialized, Role: FOLLOWER
I20260924 05:08:55.930099 2010 consensus_queue.cc:260] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:55.930243 2011 consensus_queue.cc:260] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:55.930377 2010 ts_tablet_manager.cc:1428] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:08:55.930397 1987 heartbeater.cc:503] Master 127.0.98.62:38327 was elected leader, sending a full tablet report...
I20260924 05:08:55.930497 2011 ts_tablet_manager.cc:1428] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:08:55.930611 1847 heartbeater.cc:503] Master 127.0.98.62:38327 was elected leader, sending a full tablet report...
W20260924 05:08:55.967993 1848 tablet.cc:2369] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:08:56.035652 2015 raft_consensus.cc:482] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:08:56.035761 2015 raft_consensus.cc:504] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.036079 2015 leader_election.cc:288] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fcc2875ea3c42bd882631710ccbf2e0 (127.0.98.1:40477), 62fa95f0b9984510b91f3e2c062e2bcb (127.0.98.3:45095)
I20260924 05:08:56.038934 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.036291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52909 (local address 127.0.98.1:40477)
0924 05:08:56.036333 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.036335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.036369 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:08:56.036454 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.036455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.036467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.036529 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:56.036721 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.037248 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.038204 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.038566 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.038616 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.038624 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.038815 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.038849 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:08:56.038938 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.036242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40477 (local address 127.0.98.2:52909)
0924 05:08:56.036313 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:56.036325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:56.036386 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:56.036563 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:56.036565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:56.036567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:56.036646 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.036648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.037332 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.037333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.038107 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.038108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.038634 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.038635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.038715 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.038719 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:08:56.038799 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 05:08:56.038846 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 05:08:56.039075 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.036346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54199 (local address 127.0.98.3:45095)
0924 05:08:56.036390 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.036393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.036568 (+ 175us) server_negotiation.cc:407] Connection header received
0924 05:08:56.036603 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.036604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.036618 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.036686 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:56.036859 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.037314 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.038237 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.038686 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.038780 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.038790 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.038956 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.038997 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:08:56.039075 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.036250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45095 (local address 127.0.98.2:54199)
0924 05:08:56.036501 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:56.036510 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:56.036559 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:56.036721 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:56.036722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:56.036723 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:56.036792 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.036794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.037399 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.037400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.038118 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.038120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.038807 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.038809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.038876 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.038879 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:08:56.038939 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 05:08:56.038981 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":91,"thread_start_us":107,"threads_started":1}
I20260924 05:08:56.039279 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a562880f19144b0cb2c9e3abf028a7ea" candidate_uuid: "6252d06350074e53a1d1998165d70bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" is_pre_election: true
I20260924 05:08:56.039388 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a562880f19144b0cb2c9e3abf028a7ea" candidate_uuid: "6252d06350074e53a1d1998165d70bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" is_pre_election: true
I20260924 05:08:56.039435 1661 raft_consensus.cc:2454] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6252d06350074e53a1d1998165d70bdb in term 0.
I20260924 05:08:56.039510 1941 raft_consensus.cc:2454] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6252d06350074e53a1d1998165d70bdb in term 0.
I20260924 05:08:56.039638 1732 leader_election.cc:302] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6252d06350074e53a1d1998165d70bdb, 8fcc2875ea3c42bd882631710ccbf2e0; no voters:
I20260924 05:08:56.039789 2015 raft_consensus.cc:2781] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:08:56.039840 2015 raft_consensus.cc:482] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:08:56.039865 2015 raft_consensus.cc:3037] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:56.040779 2015 raft_consensus.cc:504] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.040921 2015 leader_election.cc:288] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [CANDIDATE]: Term 1 election: Requested vote from peers 8fcc2875ea3c42bd882631710ccbf2e0 (127.0.98.1:40477), 62fa95f0b9984510b91f3e2c062e2bcb (127.0.98.3:45095)
I20260924 05:08:56.041147 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a562880f19144b0cb2c9e3abf028a7ea" candidate_uuid: "6252d06350074e53a1d1998165d70bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcc2875ea3c42bd882631710ccbf2e0"
I20260924 05:08:56.041147 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a562880f19144b0cb2c9e3abf028a7ea" candidate_uuid: "6252d06350074e53a1d1998165d70bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fa95f0b9984510b91f3e2c062e2bcb"
I20260924 05:08:56.041227 1661 raft_consensus.cc:3037] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:56.041226 1941 raft_consensus.cc:3037] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:56.042042 1941 raft_consensus.cc:2454] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6252d06350074e53a1d1998165d70bdb in term 1.
I20260924 05:08:56.042042 1661 raft_consensus.cc:2454] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6252d06350074e53a1d1998165d70bdb in term 1.
I20260924 05:08:56.042212 1734 leader_election.cc:302] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6252d06350074e53a1d1998165d70bdb, 62fa95f0b9984510b91f3e2c062e2bcb; no voters:
I20260924 05:08:56.042321 2015 raft_consensus.cc:2781] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:56.042471 2015 raft_consensus.cc:686] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 1 LEADER]: Becoming Leader. State: Replica: 6252d06350074e53a1d1998165d70bdb, State: Running, Role: LEADER
I20260924 05:08:56.042541 2015 consensus_queue.cc:237] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.043298 1503 catalog_manager.cc:5495] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6252d06350074e53a1d1998165d70bdb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6252d06350074e53a1d1998165d70bdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } health_report { overall_health: UNKNOWN } } }
I20260924 05:08:56.066468 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.064023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.0.98.2:43969)
0924 05:08:56.064070 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.064072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.064081 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:08:56.064192 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.064193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.064212 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.064291 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:56.064435 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.064882 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.065665 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.065850 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.065922 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.066121 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:56.066218 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:56.066308 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.066367 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.066385 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 05:08:56.067785 1708 tablet.cc:2369] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:08:56.068037 1941 raft_consensus.cc:1261] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Refusing update from remote peer 6252d06350074e53a1d1998165d70bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:56.068051 1661 raft_consensus.cc:1261] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Refusing update from remote peer 6252d06350074e53a1d1998165d70bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:56.068378 2015 consensus_queue.cc:1035] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:56.068502 2017 consensus_queue.cc:1035] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:56.071100 2020 mvcc.cc:205] Tried to move back new op lower bound from 7332767891732324352 to 7332767891630673920. Current Snapshot: MvccSnapshot[applied={T|T < 7332767891732324352}]
I20260924 05:08:56.074146 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:08:56.074239 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:08:56.074965 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1: Sending DeleteTablet for 3 replicas of tablet a562880f19144b0cb2c9e3abf028a7ea
I20260924 05:08:56.075315 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet a562880f19144b0cb2c9e3abf028a7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36208
I20260924 05:08:56.075349 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet a562880f19144b0cb2c9e3abf028a7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55894
I20260924 05:08:56.075385 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet a562880f19144b0cb2c9e3abf028a7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34092
I20260924 05:08:56.075618 2029 tablet_replica.cc:330] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: stopping tablet replica
I20260924 05:08:56.075641 2031 tablet_replica.cc:330] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: stopping tablet replica
I20260924 05:08:56.075728 2031 raft_consensus.cc:2229] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:08:56.075724 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 05:08:56.075829 2031 raft_consensus.cc:2258] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 05:08:56.075862 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:08:56.075809 2030 tablet_replica.cc:330] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: stopping tablet replica
I20260924 05:08:56.076102 2030 raft_consensus.cc:2229] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:08:56.076236 2029 raft_consensus.cc:2229] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:08:56.076278 2031 ts_tablet_manager.cc:1905] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:56.076431 2029 raft_consensus.cc:2258] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:56.076550 2030 raft_consensus.cc:2258] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:56.077011 1921 tablet_service.cc:1467] Processing CreateTablet for tablet eff61e1e09894a6ba567f22d85251a02 (DEFAULT_TABLE table=test-table [id=f9d2c6ae990045cc9c21491305b5e4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:56.077128 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff61e1e09894a6ba567f22d85251a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:56.077507 2029 ts_tablet_manager.cc:1905] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:56.077054 2030 ts_tablet_manager.cc:1905] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:56.078414 2031 ts_tablet_manager.cc:1918] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:08:56.078510 2031 log.cc:1196] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/a562880f19144b0cb2c9e3abf028a7ea
I20260924 05:08:56.078794 2031 ts_tablet_manager.cc:1939] T a562880f19144b0cb2c9e3abf028a7ea P 8fcc2875ea3c42bd882631710ccbf2e0: Deleting consensus metadata
I20260924 05:08:56.079031 1781 tablet_service.cc:1467] Processing CreateTablet for tablet eff61e1e09894a6ba567f22d85251a02 (DEFAULT_TABLE table=test-table [id=f9d2c6ae990045cc9c21491305b5e4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:56.079144 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff61e1e09894a6ba567f22d85251a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:56.079223 1488 catalog_manager.cc:4841] TS 8fcc2875ea3c42bd882631710ccbf2e0 (127.0.98.1:40477): tablet a562880f19144b0cb2c9e3abf028a7ea (table test-table [id=c0b366b263aa4a84913d9e8a2b346255]) successfully deleted
I20260924 05:08:56.080128 2029 ts_tablet_manager.cc:1918] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:08:56.080214 2029 log.cc:1196] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/a562880f19144b0cb2c9e3abf028a7ea
I20260924 05:08:56.080348 2011 tablet_bootstrap.cc:492] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: Bootstrap starting.
I20260924 05:08:56.080494 2029 ts_tablet_manager.cc:1939] T a562880f19144b0cb2c9e3abf028a7ea P 62fa95f0b9984510b91f3e2c062e2bcb: Deleting consensus metadata
I20260924 05:08:56.080747 2011 tablet_bootstrap.cc:654] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:56.081036 2030 ts_tablet_manager.cc:1918] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:08:56.081051 1490 catalog_manager.cc:4841] TS 62fa95f0b9984510b91f3e2c062e2bcb (127.0.98.3:45095): tablet a562880f19144b0cb2c9e3abf028a7ea (table test-table [id=c0b366b263aa4a84913d9e8a2b346255]) successfully deleted
I20260924 05:08:56.081171 2030 log.cc:1196] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/a562880f19144b0cb2c9e3abf028a7ea
I20260924 05:08:56.081254 1641 tablet_service.cc:1467] Processing CreateTablet for tablet eff61e1e09894a6ba567f22d85251a02 (DEFAULT_TABLE table=test-table [id=f9d2c6ae990045cc9c21491305b5e4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:56.081362 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff61e1e09894a6ba567f22d85251a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:56.081564 2030 ts_tablet_manager.cc:1939] T a562880f19144b0cb2c9e3abf028a7ea P 6252d06350074e53a1d1998165d70bdb: Deleting consensus metadata
I20260924 05:08:56.081905 1488 catalog_manager.cc:4841] TS 6252d06350074e53a1d1998165d70bdb (127.0.98.2:43969): tablet a562880f19144b0cb2c9e3abf028a7ea (table test-table [id=c0b366b263aa4a84913d9e8a2b346255]) successfully deleted
I20260924 05:08:56.082479 2007 tablet_bootstrap.cc:492] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: Bootstrap starting.
I20260924 05:08:56.082829 2011 tablet_bootstrap.cc:492] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: No bootstrap required, opened a new log
I20260924 05:08:56.082865 2007 tablet_bootstrap.cc:654] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:56.082887 2011 ts_tablet_manager.cc:1397] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 05:08:56.083012 2011 raft_consensus.cc:348] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.083091 2011 raft_consensus.cc:374] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:56.083113 2011 raft_consensus.cc:729] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6252d06350074e53a1d1998165d70bdb, State: Initialized, Role: FOLLOWER
I20260924 05:08:56.083156 2011 consensus_queue.cc:260] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.083240 2011 ts_tablet_manager.cc:1428] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:56.083755 2007 tablet_bootstrap.cc:492] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: No bootstrap required, opened a new log
I20260924 05:08:56.083802 2007 ts_tablet_manager.cc:1397] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 05:08:56.083809 2010 tablet_bootstrap.cc:492] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: Bootstrap starting.
I20260924 05:08:56.083930 2007 raft_consensus.cc:348] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.084002 2007 raft_consensus.cc:374] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:56.084028 2007 raft_consensus.cc:729] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcc2875ea3c42bd882631710ccbf2e0, State: Initialized, Role: FOLLOWER
I20260924 05:08:56.084071 2007 consensus_queue.cc:260] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.084172 2007 ts_tablet_manager.cc:1428] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 05:08:56.084182 2010 tablet_bootstrap.cc:654] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:56.084761 2010 tablet_bootstrap.cc:492] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: No bootstrap required, opened a new log
I20260924 05:08:56.084811 2010 ts_tablet_manager.cc:1397] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:08:56.084968 2010 raft_consensus.cc:348] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.085037 2010 raft_consensus.cc:374] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:56.085058 2010 raft_consensus.cc:729] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fa95f0b9984510b91f3e2c062e2bcb, State: Initialized, Role: FOLLOWER
I20260924 05:08:56.085107 2010 consensus_queue.cc:260] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.085192 2010 ts_tablet_manager.cc:1428] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:56.097152 2015 raft_consensus.cc:482] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:08:56.097239 2015 raft_consensus.cc:504] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.097396 2015 leader_election.cc:288] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fcc2875ea3c42bd882631710ccbf2e0 (127.0.98.1:40477), 62fa95f0b9984510b91f3e2c062e2bcb (127.0.98.3:45095)
I20260924 05:08:56.097551 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff61e1e09894a6ba567f22d85251a02" candidate_uuid: "6252d06350074e53a1d1998165d70bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" is_pre_election: true
I20260924 05:08:56.097570 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff61e1e09894a6ba567f22d85251a02" candidate_uuid: "6252d06350074e53a1d1998165d70bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" is_pre_election: true
I20260924 05:08:56.097626 1661 raft_consensus.cc:2454] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6252d06350074e53a1d1998165d70bdb in term 0.
I20260924 05:08:56.097652 1941 raft_consensus.cc:2454] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6252d06350074e53a1d1998165d70bdb in term 0.
I20260924 05:08:56.097754 1732 leader_election.cc:302] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6252d06350074e53a1d1998165d70bdb, 8fcc2875ea3c42bd882631710ccbf2e0; no voters:
I20260924 05:08:56.097860 2015 raft_consensus.cc:2781] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:08:56.097909 2015 raft_consensus.cc:482] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:08:56.097930 2015 raft_consensus.cc:3037] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:56.098691 2015 raft_consensus.cc:504] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.098858 2015 leader_election.cc:288] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [CANDIDATE]: Term 1 election: Requested vote from peers 8fcc2875ea3c42bd882631710ccbf2e0 (127.0.98.1:40477), 62fa95f0b9984510b91f3e2c062e2bcb (127.0.98.3:45095)
I20260924 05:08:56.099023 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff61e1e09894a6ba567f22d85251a02" candidate_uuid: "6252d06350074e53a1d1998165d70bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fa95f0b9984510b91f3e2c062e2bcb"
I20260924 05:08:56.099030 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff61e1e09894a6ba567f22d85251a02" candidate_uuid: "6252d06350074e53a1d1998165d70bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcc2875ea3c42bd882631710ccbf2e0"
I20260924 05:08:56.099089 1941 raft_consensus.cc:3037] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:56.099107 1661 raft_consensus.cc:3037] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:56.099774 1661 raft_consensus.cc:2454] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6252d06350074e53a1d1998165d70bdb in term 1.
I20260924 05:08:56.099792 1941 raft_consensus.cc:2454] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6252d06350074e53a1d1998165d70bdb in term 1.
I20260924 05:08:56.099929 1732 leader_election.cc:302] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6252d06350074e53a1d1998165d70bdb, 8fcc2875ea3c42bd882631710ccbf2e0; no voters:
I20260924 05:08:56.100061 2015 raft_consensus.cc:2781] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:56.100126 2015 raft_consensus.cc:686] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 1 LEADER]: Becoming Leader. State: Replica: 6252d06350074e53a1d1998165d70bdb, State: Running, Role: LEADER
I20260924 05:08:56.100167 2015 consensus_queue.cc:237] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.100724 1504 catalog_manager.cc:5495] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6252d06350074e53a1d1998165d70bdb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6252d06350074e53a1d1998165d70bdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } health_report { overall_health: UNKNOWN } } }
W20260924 05:08:56.123792 1988 tablet.cc:2369] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 05:08:56 dist-test-slave-mr2f krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:56.134799 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.127543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.0.98.62:38327)
0924 05:08:56.127590 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.127592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.127709 (+ 117us) server_negotiation.cc:407] Connection header received
0924 05:08:56.127781 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.127782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.127790 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.127840 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:56.128415 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.128889 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.129899 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.130064 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.130110 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.132249 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:56.132251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:56.132251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:56.132257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:56.133618 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:56.133989 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:56.133990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:56.133991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:56.134048 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:56.134235 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:56.134236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:56.134236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:56.134339 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:56.134418 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.134567 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.134665 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:08:56.134917 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.127454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.0.1:52444)
0924 05:08:56.127623 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:56.127635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:56.127718 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:56.127863 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:56.127864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:56.128276 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:56.128338 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.128341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.128963 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.128964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.129818 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.129821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.130132 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.130133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.130187 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.130717 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:56.130723 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:56.132162 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:56.133712 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:56.133720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:56.133722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:56.133915 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:56.134113 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:56.134114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:56.134114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:56.134174 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:56.134408 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:56.134415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:56.134459 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 05:08:56.134545 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 05:08:56.134739 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":99,"thread_start_us":86,"threads_started":1}
I20260924 05:08:56.136723 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52444:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 05:08:56.136926 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:08:56.138317 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 6c293bd86da14e1cb48ef92d1cd02108 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bdea815a68c240af9ae32d3e966a3128]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:08:56.138353 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 6c293bd86da14e1cb48ef92d1cd02108 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bdea815a68c240af9ae32d3e966a3128]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:08:56.138470 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c293bd86da14e1cb48ef92d1cd02108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:56.138478 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c293bd86da14e1cb48ef92d1cd02108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:56.138886 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 6c293bd86da14e1cb48ef92d1cd02108 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bdea815a68c240af9ae32d3e966a3128]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:08:56.139025 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c293bd86da14e1cb48ef92d1cd02108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:56.139986 2011 tablet_bootstrap.cc:492] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: Bootstrap starting.
I20260924 05:08:56.140311 2011 tablet_bootstrap.cc:654] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:56.140599 2007 tablet_bootstrap.cc:492] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: Bootstrap starting.
I20260924 05:08:56.140606 2010 tablet_bootstrap.cc:492] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: Bootstrap starting.
I20260924 05:08:56.141017 2010 tablet_bootstrap.cc:654] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:56.141037 2007 tablet_bootstrap.cc:654] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:56.141675 2007 tablet_bootstrap.cc:492] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: No bootstrap required, opened a new log
I20260924 05:08:56.141733 2007 ts_tablet_manager.cc:1397] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 05:08:56.141881 2010 tablet_bootstrap.cc:492] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: No bootstrap required, opened a new log
I20260924 05:08:56.141884 2007 raft_consensus.cc:348] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.141942 2010 ts_tablet_manager.cc:1397] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:08:56.141958 2007 raft_consensus.cc:374] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:56.141980 2007 raft_consensus.cc:729] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcc2875ea3c42bd882631710ccbf2e0, State: Initialized, Role: FOLLOWER
I20260924 05:08:56.142032 2007 consensus_queue.cc:260] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.142122 2007 ts_tablet_manager.cc:1428] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:56.142094 2010 raft_consensus.cc:348] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.142163 2010 raft_consensus.cc:374] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:56.142190 2010 raft_consensus.cc:729] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fa95f0b9984510b91f3e2c062e2bcb, State: Initialized, Role: FOLLOWER
I20260924 05:08:56.142251 2010 consensus_queue.cc:260] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.142272 2009 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.142339 2010 ts_tablet_manager.cc:1428] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:56.142349 2009 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:56.142462 2014 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.142561 2014 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:56.143231 2011 tablet_bootstrap.cc:492] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: No bootstrap required, opened a new log
I20260924 05:08:56.143297 2011 ts_tablet_manager.cc:1397] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 05:08:56.143435 2011 raft_consensus.cc:348] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.143505 2011 raft_consensus.cc:374] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:56.143537 2011 raft_consensus.cc:729] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6252d06350074e53a1d1998165d70bdb, State: Initialized, Role: FOLLOWER
I20260924 05:08:56.143587 2011 consensus_queue.cc:260] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.143673 2011 ts_tablet_manager.cc:1428] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:56.143694 2015 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.143792 2015 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:56.344349 2009 raft_consensus.cc:482] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:08:56.344468 2009 raft_consensus.cc:504] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.344803 2009 leader_election.cc:288] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6252d06350074e53a1d1998165d70bdb (127.0.98.2:43969), 62fa95f0b9984510b91f3e2c062e2bcb (127.0.98.3:45095)
I20260924 05:08:56.347909 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.345091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59691 (local address 127.0.98.3:45095)
0924 05:08:56.345143 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.345145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.345213 (+ 68us) server_negotiation.cc:407] Connection header received
0924 05:08:56.345278 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.345279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.345290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.345372 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:56.345535 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.346038 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.347213 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.347564 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.347617 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.347663 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.347789 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.347828 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:08:56.347929 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.345083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59739 (local address 127.0.98.2:43969)
0924 05:08:56.345147 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.345149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.345213 (+ 64us) server_negotiation.cc:407] Connection header received
0924 05:08:56.345334 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.345337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.345349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.345403 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:56.345619 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.346083 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.347211 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.347572 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.347656 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.347667 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.347817 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.347851 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:08:56.347946 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.345001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45095 (local address 127.0.98.1:59691)
0924 05:08:56.345134 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:56.345146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:56.345217 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:56.345376 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:56.345377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:56.345379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:56.345469 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.345472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.346136 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.346137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.347128 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.347130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.347644 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.347645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.347722 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.347725 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:08:56.347778 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 05:08:56.347820 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":42,"spinlock_wait_cycles":768,"thread_start_us":67,"threads_started":1}
I20260924 05:08:56.347955 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.345001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43969 (local address 127.0.98.1:59739)
0924 05:08:56.345134 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:56.345146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:56.345230 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:56.345426 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:56.345427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:56.345429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:56.345531 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.345533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.346193 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.346194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.347128 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.347131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.347672 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.347673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.347741 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.347745 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:08:56.347809 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 05:08:56.347843 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
I20260924 05:08:56.348167 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c293bd86da14e1cb48ef92d1cd02108" candidate_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" is_pre_election: true
I20260924 05:08:56.348268 1941 raft_consensus.cc:2454] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fcc2875ea3c42bd882631710ccbf2e0 in term 0.
I20260924 05:08:56.348376 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c293bd86da14e1cb48ef92d1cd02108" candidate_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6252d06350074e53a1d1998165d70bdb" is_pre_election: true
I20260924 05:08:56.348441 1594 leader_election.cc:302] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fa95f0b9984510b91f3e2c062e2bcb, 8fcc2875ea3c42bd882631710ccbf2e0; no voters:
I20260924 05:08:56.348495 1801 raft_consensus.cc:2454] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fcc2875ea3c42bd882631710ccbf2e0 in term 0.
I20260924 05:08:56.348555 2009 raft_consensus.cc:2781] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:08:56.348587 2009 raft_consensus.cc:482] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:08:56.348613 2009 raft_consensus.cc:3037] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:56.349339 2009 raft_consensus.cc:504] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.349498 2009 leader_election.cc:288] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6252d06350074e53a1d1998165d70bdb (127.0.98.2:43969), 62fa95f0b9984510b91f3e2c062e2bcb (127.0.98.3:45095)
I20260924 05:08:56.349684 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c293bd86da14e1cb48ef92d1cd02108" candidate_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fa95f0b9984510b91f3e2c062e2bcb"
I20260924 05:08:56.349677 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c293bd86da14e1cb48ef92d1cd02108" candidate_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6252d06350074e53a1d1998165d70bdb"
I20260924 05:08:56.349751 1801 raft_consensus.cc:3037] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:56.349751 1941 raft_consensus.cc:3037] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:56.350467 1941 raft_consensus.cc:2454] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fcc2875ea3c42bd882631710ccbf2e0 in term 1.
I20260924 05:08:56.350572 1801 raft_consensus.cc:2454] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fcc2875ea3c42bd882631710ccbf2e0 in term 1.
I20260924 05:08:56.350627 1594 leader_election.cc:302] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fa95f0b9984510b91f3e2c062e2bcb, 8fcc2875ea3c42bd882631710ccbf2e0; no voters:
I20260924 05:08:56.350709 2009 raft_consensus.cc:2781] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:56.350904 2009 raft_consensus.cc:686] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8fcc2875ea3c42bd882631710ccbf2e0, State: Running, Role: LEADER
I20260924 05:08:56.350996 2009 consensus_queue.cc:237] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } }
I20260924 05:08:56.351325 2047 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: TxnStatusTablet state changed. Reason: New leader 8fcc2875ea3c42bd882631710ccbf2e0. Latest consensus state: current_term: 1 leader_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.351437 2047 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:08:56.351693 1504 catalog_manager.cc:5495] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fcc2875ea3c42bd882631710ccbf2e0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } health_report { overall_health: UNKNOWN } } }
I20260924 05:08:56.352061 2049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 05:08:56.352185 2049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 05:08:56.371416 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.367553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40477 (local address 127.0.0.1:34104)
0924 05:08:56.367636 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:56.367647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:56.367688 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:56.368314 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:56.368315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:56.368316 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:08:56.368386 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.368388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.369320 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.369321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.370285 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.370287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.370744 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.370745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.370835 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.370857 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:08:56.371183 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:08:56.371193 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 05:08:56.371274 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 05:08:56.371321 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 05:08:56.371749 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.368057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.0.98.1:40477)
0924 05:08:56.368112 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.368114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.368124 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:56.368178 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.368180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.368202 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.368287 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:56.368462 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.369221 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.370362 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.370652 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.370741 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.370971 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:56.371092 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:56.371196 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.371644 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.371667 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:08:56.372864 1801 raft_consensus.cc:1261] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb [term 1 FOLLOWER]: Refusing update from remote peer 8fcc2875ea3c42bd882631710ccbf2e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:56.373018 1941 raft_consensus.cc:1261] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Refusing update from remote peer 8fcc2875ea3c42bd882631710ccbf2e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:56.373131 2047 consensus_queue.cc:1035] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:56.373234 2009 consensus_queue.cc:1035] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:56.373934 2014 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: TxnStatusTablet state changed. Reason: New leader 8fcc2875ea3c42bd882631710ccbf2e0. Latest consensus state: current_term: 1 leader_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.374039 2014 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:56.374142 2015 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: TxnStatusTablet state changed. Reason: New leader 8fcc2875ea3c42bd882631710ccbf2e0. Latest consensus state: current_term: 1 leader_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.374221 2015 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:56.374850 2009 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.374966 2009 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:08:56.375418 2015 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.375494 2015 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 6252d06350074e53a1d1998165d70bdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:56.376222 2009 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.376307 2009 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 8fcc2875ea3c42bd882631710ccbf2e0: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:08:56.376467 2014 tablet_replica.cc:439] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 } } peers { permanent_uuid: "6252d06350074e53a1d1998165d70bdb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } } peers { permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 } } }
I20260924 05:08:56.376533 2014 tablet_replica.cc:442] T 6c293bd86da14e1cb48ef92d1cd02108 P 62fa95f0b9984510b91f3e2c062e2bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:56.381136 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.378692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58509 (local address 127.0.98.1:40477)
0924 05:08:56.378736 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.378738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.378812 (+ 74us) server_negotiation.cc:407] Connection header received
0924 05:08:56.378878 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.378878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.378886 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.378925 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:56.379040 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.379494 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.380581 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.380750 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.380797 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.380968 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:56.380990 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:56.381026 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.381078 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.381091 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:08:56.381349 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.378612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40477 (local address 127.0.98.2:58509)
0924 05:08:56.378731 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:56.378743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:56.378827 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:56.378937 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:56.378937 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:56.378939 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:08:56.378987 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.378989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.379565 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.379566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.380499 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.380501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.380818 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.380819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.380877 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.380891 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:08:56.381027 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:08:56.381028 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:08:56.381077 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:08:56.381120 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":67,"threads_started":1}
I20260924 05:08:56.383455 1941 raft_consensus.cc:1261] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Refusing update from remote peer 6252d06350074e53a1d1998165d70bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:56.383466 1661 raft_consensus.cc:1261] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Refusing update from remote peer 6252d06350074e53a1d1998165d70bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:56.383688 2017 consensus_queue.cc:1035] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "62fa95f0b9984510b91f3e2c062e2bcb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:56.383849 2017 consensus_queue.cc:1035] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcc2875ea3c42bd882631710ccbf2e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:56.391955 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.389147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39759 (local address 127.0.98.2:43969)
0924 05:08:56.389181 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.389183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.389304 (+ 121us) server_negotiation.cc:407] Connection header received
0924 05:08:56.389403 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.389404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.389415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.389443 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:56.389648 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.390073 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.391216 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.391509 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.391547 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.391766 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:56.391794 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:56.391836 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.391888 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.391905 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:08:56.392019 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.389087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43969 (local address 127.0.98.1:39759)
0924 05:08:56.389248 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:56.389258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:56.389326 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:56.389499 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:56.389500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:56.389502 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:08:56.389569 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.389570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.390150 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.390151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.391120 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:56.391122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.391597 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.391598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:56.391658 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.391672 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:08:56.391836 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:08:56.391838 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:08:56.391888 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:08:56.391926 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 05:08:56.395717 2022 mvcc.cc:205] Tried to move back new op lower bound from 7332767893025267712 to 7332767891866546176. Current Snapshot: MvccSnapshot[applied={T|T < 7332767893025267712}]
I20260924 05:08:56.400664 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:08:56.400775 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:08:56.401388 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2aa0952bf604aeaad9a0f480b0004c1: Sending DeleteTablet for 3 replicas of tablet eff61e1e09894a6ba567f22d85251a02
I20260924 05:08:56.401664 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet eff61e1e09894a6ba567f22d85251a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36208
I20260924 05:08:56.401664 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet eff61e1e09894a6ba567f22d85251a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55894
I20260924 05:08:56.401664 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet eff61e1e09894a6ba567f22d85251a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34092
I20260924 05:08:56.401757 2031 tablet_replica.cc:330] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: stopping tablet replica
I20260924 05:08:56.401758 2030 tablet_replica.cc:330] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: stopping tablet replica
I20260924 05:08:56.401767 2029 tablet_replica.cc:330] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: stopping tablet replica
I20260924 05:08:56.401799 2031 raft_consensus.cc:2229] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:08:56.401814 2029 raft_consensus.cc:2229] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:08:56.401811 2030 raft_consensus.cc:2229] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:08:56.401859 2031 raft_consensus.cc:2258] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:56.401859 2029 raft_consensus.cc:2258] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:56.401863 2030 raft_consensus.cc:2258] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:56.402161 2029 ts_tablet_manager.cc:1905] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:56.402217 2031 ts_tablet_manager.cc:1905] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:56.403653 2031 ts_tablet_manager.cc:1918] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:08:56.403666 2029 ts_tablet_manager.cc:1918] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:08:56.403680 2030 ts_tablet_manager.cc:1905] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:56.403715 2031 log.cc:1196] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/eff61e1e09894a6ba567f22d85251a02
I20260924 05:08:56.403724 2029 log.cc:1196] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/eff61e1e09894a6ba567f22d85251a02
I20260924 05:08:56.403941 2029 ts_tablet_manager.cc:1939] T eff61e1e09894a6ba567f22d85251a02 P 62fa95f0b9984510b91f3e2c062e2bcb: Deleting consensus metadata
I20260924 05:08:56.403941 2031 ts_tablet_manager.cc:1939] T eff61e1e09894a6ba567f22d85251a02 P 8fcc2875ea3c42bd882631710ccbf2e0: Deleting consensus metadata
I20260924 05:08:56.404223 1488 catalog_manager.cc:4841] TS 8fcc2875ea3c42bd882631710ccbf2e0 (127.0.98.1:40477): tablet eff61e1e09894a6ba567f22d85251a02 (table test-table [id=f9d2c6ae990045cc9c21491305b5e4fc]) successfully deleted
I20260924 05:08:56.404219 1490 catalog_manager.cc:4841] TS 62fa95f0b9984510b91f3e2c062e2bcb (127.0.98.3:45095): tablet eff61e1e09894a6ba567f22d85251a02 (table test-table [id=f9d2c6ae990045cc9c21491305b5e4fc]) successfully deleted
I20260924 05:08:56.405131 2030 ts_tablet_manager.cc:1918] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:08:56.405179 2030 log.cc:1196] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/eff61e1e09894a6ba567f22d85251a02
I20260924 05:08:56.405388 2030 ts_tablet_manager.cc:1939] T eff61e1e09894a6ba567f22d85251a02 P 6252d06350074e53a1d1998165d70bdb: Deleting consensus metadata
I20260924 05:08:56.405782 1488 catalog_manager.cc:4841] TS 6252d06350074e53a1d1998165d70bdb (127.0.98.2:43969): tablet eff61e1e09894a6ba567f22d85251a02 (table test-table [id=f9d2c6ae990045cc9c21491305b5e4fc]) successfully deleted
Sep 24 05:08:56 dist-test-slave-mr2f krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:08:56.420619 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.411053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.0.98.1:40477)
0924 05:08:56.411113 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.411115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.411189 (+ 74us) server_negotiation.cc:407] Connection header received
0924 05:08:56.411264 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.411265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.411275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.411330 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:56.412017 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.412777 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.413475 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.413771 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.413842 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.416724 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:56.416733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:56.416734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:56.416741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:56.419427 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:56.419768 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:56.419772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:56.419772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:56.419837 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:56.420016 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:56.420016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:56.420017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:56.420199 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:56.420346 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.420418 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.420497 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 05:08:56.420900 1687 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34120
I20260924 05:08:56.428007 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:56.421903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.0.98.62:38327)
0924 05:08:56.421947 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:56.421948 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:56.421995 (+ 47us) server_negotiation.cc:407] Connection header received
0924 05:08:56.422062 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:56.422063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:56.422071 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:56.422128 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:56.422720 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.423239 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.424234 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:56.424406 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:56.424454 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:56.425682 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:56.425683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:56.425683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:56.425688 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:56.427025 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:56.427381 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:56.427381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:56.427382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:56.427426 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:56.427610 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:56.427611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:56.427611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:56.427696 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:56.427771 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:56.427896 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 05:08:56.427948 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 05:08:56.428215 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52454
I20260924 05:08:56.428958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 1576
I20260924 05:08:56.434994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 1716
I20260924 05:08:56.440791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 1856
I20260924 05:08:56.445402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 1472
2026-09-24T05:08:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2330 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:08:56 dist-test-slave-mr2f krb5kdc[2070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:08:56 dist-test-slave-mr2f krb5kdc[2070](info): set up 2 sockets
Sep 24 05:08:56 dist-test-slave-mr2f krb5kdc[2070](info): commencing operation
krb5kdc: starting...
W20260924 05:08:57.454581 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:08:57 dist-test-slave-mr2f krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:08:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:08:57.563696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33257
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:57.640501 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:57.640722 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:57.640743 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:57.641767 2083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:08:57.641798 2083 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 05:08:57.641811 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:57.641824 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:57.641835 2083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:08:57.641847 2083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:08:57.643193 2083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33257
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:57.643368 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:57.643532 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:57.648000 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:57.648067 2083 server_base.cc:1034] running on GCE node
W20260924 05:08:57.648028 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:57.648036 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:57.648380 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:57.648675 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:57.649809 2083 hybrid_clock.cc:648] HybridClock initialized: now 1790226537649800 us; error 27 us; skew 500 ppm
Sep 24 05:08:57 dist-test-slave-mr2f krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:57.652209 2083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:08:57.652633 2083 webserver.cc:460] Webserver started at http://127.0.98.62:41089/ using document root <none> and password file <none>
I20260924 05:08:57.652855 2083 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:57.652896 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:57.653031 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:57.654006 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "1d869ebe89774cccb0878596fbdff2cf"
format_stamp: "Formatted at 2026-09-24 05:08:57 on dist-test-slave-mr2f"
server_key: "398b7a450667b9b64a571fb3233fb8ee"
server_key_iv: "5152206c650814d0fe76b2cfe4b2bfb9"
server_key_version: "encryptionkey@0"
I20260924 05:08:57.654347 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d869ebe89774cccb0878596fbdff2cf"
format_stamp: "Formatted at 2026-09-24 05:08:57 on dist-test-slave-mr2f"
server_key: "398b7a450667b9b64a571fb3233fb8ee"
server_key_iv: "5152206c650814d0fe76b2cfe4b2bfb9"
server_key_version: "encryptionkey@0"
I20260924 05:08:57.655692 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:08:57.656414 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:57.656625 2083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:08:57.656694 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "1d869ebe89774cccb0878596fbdff2cf"
format_stamp: "Formatted at 2026-09-24 05:08:57 on dist-test-slave-mr2f"
server_key: "398b7a450667b9b64a571fb3233fb8ee"
server_key_iv: "5152206c650814d0fe76b2cfe4b2bfb9"
server_key_version: "encryptionkey@0"
I20260924 05:08:57.656771 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:57.669433 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:57.671145 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:57.671284 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:57.675662 2083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33257
I20260924 05:08:57.675709 2160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33257 every 8 connection(s)
I20260924 05:08:57.676045 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:08:57.676647 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:57.677634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 2083
I20260924 05:08:57.677732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:08:57.677881 392 external_mini_cluster.cc:1428] Setting key 13a1506f2c4d939c607d3599091592c4
I20260924 05:08:57.679382 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf: Bootstrap starting.
I20260924 05:08:57.680016 2161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:57.680322 2161 log.cc:824] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:57.681071 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf: No bootstrap required, opened a new log
I20260924 05:08:57.683239 2161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d869ebe89774cccb0878596fbdff2cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33257 } }
I20260924 05:08:57.683377 2161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:57.683400 2161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d869ebe89774cccb0878596fbdff2cf, State: Initialized, Role: FOLLOWER
I20260924 05:08:57.683538 2161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d869ebe89774cccb0878596fbdff2cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33257 } }
I20260924 05:08:57.683624 2161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:08:57.683646 2161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:08:57.683710 2161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:57.684195 2161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d869ebe89774cccb0878596fbdff2cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33257 } }
I20260924 05:08:57.684320 2161 leader_election.cc:302] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d869ebe89774cccb0878596fbdff2cf; no voters:
I20260924 05:08:57.684485 2161 leader_election.cc:288] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:08:57.684543 2166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:57.684723 2166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [term 1 LEADER]: Becoming Leader. State: Replica: 1d869ebe89774cccb0878596fbdff2cf, State: Running, Role: LEADER
I20260924 05:08:57.684731 2161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:08:57.684837 2166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d869ebe89774cccb0878596fbdff2cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33257 } }
Sep 24 05:08:57 dist-test-slave-mr2f krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:57.685119 2166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d869ebe89774cccb0878596fbdff2cf. Latest consensus state: current_term: 1 leader_uuid: "1d869ebe89774cccb0878596fbdff2cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d869ebe89774cccb0878596fbdff2cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33257 } } }
I20260924 05:08:57.685210 2166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:57.685129 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d869ebe89774cccb0878596fbdff2cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d869ebe89774cccb0878596fbdff2cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33257 } } }
I20260924 05:08:57.685304 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf [sys.catalog]: This master's current role is: LEADER
I20260924 05:08:57.686328 2172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:08:57.687091 2172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:08:57.688961 2172 catalog_manager.cc:1347] Generated new cluster ID: d4eff739961140f3a84767e81a877821
I20260924 05:08:57.689020 2172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:08:57.689169 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:57.678377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.0.98.62:33257)
0924 05:08:57.678558 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:57.678568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:57.678599 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:08:57.679135 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:57.679144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:57.679203 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:57.679353 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:57.679924 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.681354 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.682067 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.682693 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.682840 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:57.685492 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:57.685497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:57.685498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:57.685513 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:57.687543 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:57.687878 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:57.687881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:57.687881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:57.687982 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:57.688226 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:57.688227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:57.688227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:57.688500 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:57.688635 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:57.688810 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 05:08:57.688955 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260924 05:08:57.692044 2172 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:08:57.692695 2172 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:08:57.702227 2172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf: Generated new TSK 0
I20260924 05:08:57.702381 2172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:08:57.729723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33257
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:57.807248 2187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:57.807459 2187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:57.807499 2187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:57.808692 2187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:57.808758 2187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:57.808789 2187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 05:08:57.808818 2187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:08:57.808854 2187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:08:57.810215 2187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:57.810461 2187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:57.810674 2187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:08:57.815579 2187 server_base.cc:1034] running on GCE node
W20260924 05:08:57.815507 2193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:57.815507 2192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:57.815496 2195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:57.815932 2187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:57.816171 2187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:57.817329 2187 hybrid_clock.cc:648] HybridClock initialized: now 1790226537817318 us; error 25 us; skew 500 ppm
Sep 24 05:08:57 dist-test-slave-mr2f krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:57.819707 2187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:08:57.820063 2187 webserver.cc:460] Webserver started at http://127.0.98.1:34135/ using document root <none> and password file <none>
I20260924 05:08:57.820227 2187 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:57.820274 2187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:57.820366 2187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:57.821295 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1fcae7ed09d4fa6986255a87a28d5f2"
format_stamp: "Formatted at 2026-09-24 05:08:57 on dist-test-slave-mr2f"
server_key: "ff0c90d373e6b45bf0c6f1f463c96a30"
server_key_iv: "a5385a581d83a0e0ad4bc650a9558a2b"
server_key_version: "encryptionkey@0"
I20260924 05:08:57.821587 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1fcae7ed09d4fa6986255a87a28d5f2"
format_stamp: "Formatted at 2026-09-24 05:08:57 on dist-test-slave-mr2f"
server_key: "ff0c90d373e6b45bf0c6f1f463c96a30"
server_key_iv: "a5385a581d83a0e0ad4bc650a9558a2b"
server_key_version: "encryptionkey@0"
I20260924 05:08:57.822822 2187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:08:57.823485 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:57.823706 2187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:08:57.823807 2187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "e1fcae7ed09d4fa6986255a87a28d5f2"
format_stamp: "Formatted at 2026-09-24 05:08:57 on dist-test-slave-mr2f"
server_key: "ff0c90d373e6b45bf0c6f1f463c96a30"
server_key_iv: "a5385a581d83a0e0ad4bc650a9558a2b"
server_key_version: "encryptionkey@0"
I20260924 05:08:57.823890 2187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:57.862687 2187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:57.864681 2187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:57.864861 2187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:57.865381 2187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:57.865446 2187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:57.865485 2187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:57.865533 2187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:57.872123 2187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33043
I20260924 05:08:57.872202 2317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33043 every 8 connection(s)
I20260924 05:08:57.872504 2187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:08:57 dist-test-slave-mr2f krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:57.874387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 2187
I20260924 05:08:57.874495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:08:57.874691 392 external_mini_cluster.cc:1428] Setting key d526baf959cc9e71daecdbde49e3401a
I20260924 05:08:57.875691 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:57.865922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57279 (local address 127.0.98.62:33257)
0924 05:08:57.865993 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:57.865996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:57.866186 (+ 190us) server_negotiation.cc:407] Connection header received
0924 05:08:57.866816 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:57.866818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:57.866833 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:57.866906 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:57.867759 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.868606 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.870214 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.870573 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.870659 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:57.873051 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:57.873053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:57.873053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:57.873059 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:57.874169 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:57.874643 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:57.874644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:57.874644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:57.874693 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:57.875013 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:57.875014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:57.875014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:57.875158 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:57.875231 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:57.875561 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 05:08:57.875628 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:08:57.875988 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:57.865899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33257 (local address 127.0.98.1:57279)
0924 05:08:57.866076 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:57.866115 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:57.866654 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:57.867021 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:57.867025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:57.867389 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:57.867669 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:57.867674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.868723 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.868724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:57.870105 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:57.870108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.870676 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.870677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:57.870835 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:57.871367 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:57.871376 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:57.872973 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:57.874257 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:57.874264 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:57.874266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:57.874566 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:57.874755 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:57.874756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:57.874756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:57.874918 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:57.875237 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:57.875241 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:57.875422 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 05:08:57.875540 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 05:08:57.875706 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
I20260924 05:08:57.876500 2318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33257
I20260924 05:08:57.876603 2318 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:57.876806 2318 heartbeater.cc:511] Master 127.0.98.62:33257 requested a full tablet report, sending...
I20260924 05:08:57.877323 2114 ts_manager.cc:194] Registered new tserver with Master: e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043)
I20260924 05:08:57.878261 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57279
I20260924 05:08:57.883558 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:57.877684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48807 (local address 127.0.98.62:33257)
0924 05:08:57.877722 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:57.877724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:57.877732 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:08:57.877801 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:57.877802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:57.877812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:57.877867 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:57.878173 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.878876 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.879954 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.880126 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.880181 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:57.881181 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:57.881183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:57.881183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:57.881188 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:57.882501 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:57.882917 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:57.882918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:57.882918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:57.882965 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:57.883140 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:57.883141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:57.883141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:57.883228 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:57.883290 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:57.883447 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 05:08:57.883500 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:08:57.883671 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:57.877510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33257 (local address 127.0.98.1:48807)
0924 05:08:57.877673 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:57.877684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:57.877750 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:57.877870 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:57.877870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:57.878051 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:57.878107 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:57.878109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.878969 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.878970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:57.879857 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:57.879859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:57.880203 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:57.880204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:57.880285 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:57.880662 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:57.880668 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:57.881086 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:57.882583 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:57.882585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:57.882585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:57.882844 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:57.883017 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:57.883017 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:57.883018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:57.883082 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:57.883309 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:57.883310 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:57.883356 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:08:57.883430 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 05:08:57.883539 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:08:57.899632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33257
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:57.978111 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:57.978322 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:57.978363 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:57.979540 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:57.979606 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:57.979643 2327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 05:08:57.979689 2327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:08:57.979728 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:08:57.981156 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:57.981423 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:57.981660 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:08:57.986785 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:57.986792 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:57.986796 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:57.986992 2327 server_base.cc:1034] running on GCE node
I20260924 05:08:57.987207 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:57.987404 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:57.988534 2327 hybrid_clock.cc:648] HybridClock initialized: now 1790226537988526 us; error 26 us; skew 500 ppm
Sep 24 05:08:57 dist-test-slave-mr2f krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:57.990904 2327 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:08:57.991269 2327 webserver.cc:460] Webserver started at http://127.0.98.2:39621/ using document root <none> and password file <none>
I20260924 05:08:57.991463 2327 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:57.991503 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:57.991591 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:57.992549 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa43c6af00ed46b49b1794ca3484a2fb"
format_stamp: "Formatted at 2026-09-24 05:08:57 on dist-test-slave-mr2f"
server_key: "20e84b22124dcf629a6a8fbf6a60dce5"
server_key_iv: "73da9f2abc820f41f00dbd7f623688ca"
server_key_version: "encryptionkey@0"
I20260924 05:08:57.992844 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa43c6af00ed46b49b1794ca3484a2fb"
format_stamp: "Formatted at 2026-09-24 05:08:57 on dist-test-slave-mr2f"
server_key: "20e84b22124dcf629a6a8fbf6a60dce5"
server_key_iv: "73da9f2abc820f41f00dbd7f623688ca"
server_key_version: "encryptionkey@0"
I20260924 05:08:57.994058 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:08:57.994699 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:57.994871 2327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:57.994949 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "fa43c6af00ed46b49b1794ca3484a2fb"
format_stamp: "Formatted at 2026-09-24 05:08:57 on dist-test-slave-mr2f"
server_key: "20e84b22124dcf629a6a8fbf6a60dce5"
server_key_iv: "73da9f2abc820f41f00dbd7f623688ca"
server_key_version: "encryptionkey@0"
I20260924 05:08:57.995002 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:58.013687 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:58.015545 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:58.015650 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:58.016075 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:58.016115 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:58.016139 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:58.016177 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 05:08:58 dist-test-slave-mr2f krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:58.023088 2327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46389
I20260924 05:08:58.023537 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:08:58.023814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 2327
I20260924 05:08:58.023905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:08:58.024073 392 external_mini_cluster.cc:1428] Setting key 0ac261083867e548b040a595404af6cf
I20260924 05:08:58.024369 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46389 every 8 connection(s)
I20260924 05:08:58.028076 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.016555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58777 (local address 127.0.98.62:33257)
0924 05:08:58.016627 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.016630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.016805 (+ 175us) server_negotiation.cc:407] Connection header received
0924 05:08:58.017274 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.017275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.017288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.017370 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:58.018205 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.019057 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.020674 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.021027 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.021103 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.024375 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:58.024376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:58.024377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:58.024385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:58.026528 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.027060 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.027061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.027062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.027135 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.027403 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.027404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.027405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.027518 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:58.027593 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.027921 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.028015 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 05:08:58.028358 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.016525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33257 (local address 127.0.98.2:58777)
0924 05:08:58.016682 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.016720 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.017185 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.017459 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.017464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.017817 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:58.018101 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.018108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.019153 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.019154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.020561 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.020563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.021116 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.021117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.021227 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.021749 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:58.021756 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:58.023535 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:58.026648 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.026654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.026656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.026996 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.027207 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.027208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.027208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.027345 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.027584 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:58.027590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:58.027745 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.027900 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.028095 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260924 05:08:58.028858 2458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33257
I20260924 05:08:58.028932 2458 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:58.029114 2458 heartbeater.cc:511] Master 127.0.98.62:33257 requested a full tablet report, sending...
I20260924 05:08:58.029471 2115 ts_manager.cc:194] Registered new tserver with Master: fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389)
I20260924 05:08:58.029865 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58777
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 05:08:58.036710 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.030957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33257 (local address 127.0.98.2:56163)
0924 05:08:58.031095 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.031107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.031169 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.031280 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.031280 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.031500 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:58.031562 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.031564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.032171 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.032172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.033044 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.033047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.033445 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.033446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.033503 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.033835 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:58.033839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:58.034119 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:58.035591 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.035592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.035593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.035837 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.036077 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.036078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.036078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.036129 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.036473 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:58.036473 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:58.036504 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.036579 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.036644 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260924 05:08:58.036814 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.031048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56163 (local address 127.0.98.62:33257)
0924 05:08:58.031087 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.031089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.031153 (+ 64us) server_negotiation.cc:407] Connection header received
0924 05:08:58.031221 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.031222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.031229 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.031273 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:58.031632 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.032070 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.033186 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.033356 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.033420 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.034214 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:58.034216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:58.034216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:58.034223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:58.035506 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.035942 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.035943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.035943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.036012 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.036191 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.036192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.036193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.036391 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:58.036485 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.036603 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.036679 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:08:58.049347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33257
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:08:58.127626 2467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:08:58.127828 2467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:08:58.127867 2467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:08:58.129104 2467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:08:58.129163 2467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:08:58.129201 2467 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 05:08:58.129248 2467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:08:58.129294 2467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:08:58.130828 2467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:08:58.131076 2467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:08:58.131294 2467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:08:58.136165 2467 server_base.cc:1034] running on GCE node
W20260924 05:08:58.136082 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:58.136082 2473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:08:58.136127 2472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:08:58.136583 2467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:08:58.136811 2467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:08:58.137949 2467 hybrid_clock.cc:648] HybridClock initialized: now 1790226538137938 us; error 27 us; skew 500 ppm
Sep 24 05:08:58 dist-test-slave-mr2f krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:08:58.140237 2467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:08:58.140580 2467 webserver.cc:460] Webserver started at http://127.0.98.3:46675/ using document root <none> and password file <none>
I20260924 05:08:58.140755 2467 fs_manager.cc:359] Metadata directory not provided
I20260924 05:08:58.140792 2467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:08:58.140872 2467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:08:58.141767 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "a4353208709943d985c0f69eedf745fe"
format_stamp: "Formatted at 2026-09-24 05:08:58 on dist-test-slave-mr2f"
server_key: "75346525b94a5dca67f94234f686d298"
server_key_iv: "8ddf09527b91c73e6fe42ee015d288bf"
server_key_version: "encryptionkey@0"
I20260924 05:08:58.142043 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a4353208709943d985c0f69eedf745fe"
format_stamp: "Formatted at 2026-09-24 05:08:58 on dist-test-slave-mr2f"
server_key: "75346525b94a5dca67f94234f686d298"
server_key_iv: "8ddf09527b91c73e6fe42ee015d288bf"
server_key_version: "encryptionkey@0"
I20260924 05:08:58.143265 2467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:08:58.143949 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:58.144114 2467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:58.144183 2467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "a4353208709943d985c0f69eedf745fe"
format_stamp: "Formatted at 2026-09-24 05:08:58 on dist-test-slave-mr2f"
server_key: "75346525b94a5dca67f94234f686d298"
server_key_iv: "8ddf09527b91c73e6fe42ee015d288bf"
server_key_version: "encryptionkey@0"
I20260924 05:08:58.144234 2467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:08:58.154350 2467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:08:58.156085 2467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:08:58.156185 2467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:08:58.156661 2467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:08:58.156699 2467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:58.156721 2467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:08:58.156755 2467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:58.163170 2467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46861
I20260924 05:08:58.163255 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46861 every 8 connection(s)
Sep 24 05:08:58 dist-test-slave-mr2f krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:58.163547 2467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:08:58.163841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 2467
I20260924 05:08:58.163959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:08:58.164158 392 external_mini_cluster.cc:1428] Setting key 5f1e4f0f936077e04dd3681edcacf8b2
I20260924 05:08:58.166908 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.157136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38829 (local address 127.0.98.62:33257)
0924 05:08:58.157201 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.157204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.157359 (+ 155us) server_negotiation.cc:407] Connection header received
0924 05:08:58.157803 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.157805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.157816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.157895 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:58.158670 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.159504 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.161087 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.161441 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.161515 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.164119 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:58.164121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:58.164122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:58.164129 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:58.165531 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.165934 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.165936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.165936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.166014 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.166269 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.166270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.166270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.166384 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:58.166457 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.166708 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.166819 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:08:58.167069 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.157093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33257 (local address 127.0.98.3:38829)
0924 05:08:58.157242 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.157278 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.157719 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.157983 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.157987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.158306 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:58.158565 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.158570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.159626 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.159627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.160974 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.160976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.161528 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.161529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.161643 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.162142 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:58.162150 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:58.164042 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:58.165630 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.165636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.165638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.165847 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.166080 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.166081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.166081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.166189 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.166449 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:58.166454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:58.166584 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.166688 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.166854 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260924 05:08:58.167480 2598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33257
I20260924 05:08:58.167569 2598 heartbeater.cc:464] Registering TS with master...
I20260924 05:08:58.167728 2598 heartbeater.cc:511] Master 127.0.98.62:33257 requested a full tablet report, sending...
I20260924 05:08:58.168113 2115 ts_manager.cc:194] Registered new tserver with Master: a4353208709943d985c0f69eedf745fe (127.0.98.3:46861)
I20260924 05:08:58.168594 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38829
I20260924 05:08:58.173431 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.168537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60813 (local address 127.0.98.62:33257)
0924 05:08:58.168580 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.168582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.168639 (+ 57us) server_negotiation.cc:407] Connection header received
0924 05:08:58.168726 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.168727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.168736 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.168767 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:58.169092 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.169560 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.170251 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.170452 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.170535 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.171374 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:58.171375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:58.171375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:58.171381 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:58.172385 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.172716 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.172717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.172718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.172772 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.172960 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.172961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.172961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.173062 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:58.173128 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.173278 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.173345 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:08:58.173458 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.168458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33257 (local address 127.0.98.3:60813)
0924 05:08:58.168586 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.168597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.168664 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.168808 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.168808 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.168974 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:58.169018 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.169020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.169652 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.169653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.170166 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.170168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.170537 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.170538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.170608 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.171018 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:58.171022 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:58.171275 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:58.172468 (+ 1193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.172469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.172470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.172635 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.172834 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.172835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.172835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.172889 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.173131 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:58.173133 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:58.173171 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.173252 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.173347 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260924 05:08:58.175625 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:08:58 dist-test-slave-mr2f krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:08:58 dist-test-slave-mr2f krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:58.197027 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.190393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.0.98.62:33257)
0924 05:08:58.190455 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.190457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.190467 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:58.190550 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.190551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.190560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.190630 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:58.191234 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.191712 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.192453 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.192637 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.192724 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.194713 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:58.194714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:58.194715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:58.194722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:58.195917 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.196271 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.196272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.196273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.196330 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.196522 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.196523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.196524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.196632 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:58.196737 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.196853 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.196926 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:08:58.198460 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:08:58.200014 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:08:58.206537 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.202213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.0.1:46358)
0924 05:08:58.202682 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.202724 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.202823 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.203290 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.203290 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.203292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:58.203343 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.203344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.204192 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.204193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.205560 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.205562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.206246 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.206247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.206320 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.206333 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.206405 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.206456 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":11,"thread_start_us":55,"threads_started":1}
I20260924 05:08:58.206555 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.202406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.98.1:33043)
0924 05:08:58.203157 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.203161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.203171 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:58.203206 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.203208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.203223 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.203290 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:58.203395 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.204095 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.205658 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.206144 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.206242 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.206305 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.206419 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.206466 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":102,"threads_started":1}
I20260924 05:08:58.206909 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.202610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.0.98.2:46389)
0924 05:08:58.202754 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.202800 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.202840 (+ 40us) server_negotiation.cc:407] Connection header received
0924 05:08:58.202942 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.202946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.202961 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.203034 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:58.203208 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.204333 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.205525 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.206257 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.206330 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.206384 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.206736 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.206808 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260924 05:08:58.206957 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.202552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.0.98.3:46861)
0924 05:08:58.202724 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.202728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.202877 (+ 149us) server_negotiation.cc:407] Connection header received
0924 05:08:58.202953 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.202955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.202970 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.203042 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:58.203220 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.204332 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.205639 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.206261 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.206330 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.206385 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.206820 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.206857 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260924 05:08:58.207221 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.202447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46861 (local address 127.0.0.1:52402)
0924 05:08:58.202815 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.202825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.202886 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.203110 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.203111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.203112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:58.203147 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.203148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.204418 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.204418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.205560 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.205562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.206423 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.206423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.206492 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.206497 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.206544 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.207091 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":48,"thread_start_us":55,"threads_started":1}
I20260924 05:08:58.207227 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.202236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.0.1:35392)
0924 05:08:58.202756 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.202794 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.202845 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.203057 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.203060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.203062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:58.203122 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.203123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.204432 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.204433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.205442 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.205444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.206421 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.206422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.206492 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.206496 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.206544 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.207091 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260924 05:08:58.207253 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 23c71e70b6ed47809576b89ef106359d (DEFAULT_TABLE table=test-table [id=3a9b962dc5a146f9a008479de23bc13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:58.207619 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c71e70b6ed47809576b89ef106359d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:58.207894 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 23c71e70b6ed47809576b89ef106359d (DEFAULT_TABLE table=test-table [id=3a9b962dc5a146f9a008479de23bc13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:58.208206 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c71e70b6ed47809576b89ef106359d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:58.210632 2618 tablet_bootstrap.cc:492] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: Bootstrap starting.
I20260924 05:08:58.210837 2619 tablet_bootstrap.cc:492] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: Bootstrap starting.
I20260924 05:08:58.211230 2618 tablet_bootstrap.cc:654] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:58.211292 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 23c71e70b6ed47809576b89ef106359d (DEFAULT_TABLE table=test-table [id=3a9b962dc5a146f9a008479de23bc13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:58.211558 2618 log.cc:824] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:58.211627 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23c71e70b6ed47809576b89ef106359d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:58.211931 2619 tablet_bootstrap.cc:654] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:58.212248 2619 log.cc:824] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:58.212296 2618 tablet_bootstrap.cc:492] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: No bootstrap required, opened a new log
I20260924 05:08:58.212384 2618 ts_tablet_manager.cc:1397] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:08:58.213043 2619 tablet_bootstrap.cc:492] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: No bootstrap required, opened a new log
I20260924 05:08:58.213114 2619 ts_tablet_manager.cc:1397] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:08:58.214077 2618 raft_consensus.cc:348] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.214236 2618 raft_consensus.cc:374] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:58.214280 2618 raft_consensus.cc:729] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fcae7ed09d4fa6986255a87a28d5f2, State: Initialized, Role: FOLLOWER
I20260924 05:08:58.214387 2618 consensus_queue.cc:260] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.214655 2618 ts_tablet_manager.cc:1428] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:08:58.214797 2318 heartbeater.cc:503] Master 127.0.98.62:33257 was elected leader, sending a full tablet report...
I20260924 05:08:58.215193 2623 tablet_bootstrap.cc:492] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: Bootstrap starting.
I20260924 05:08:58.215289 2619 raft_consensus.cc:348] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.215440 2619 raft_consensus.cc:374] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:58.215471 2619 raft_consensus.cc:729] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4353208709943d985c0f69eedf745fe, State: Initialized, Role: FOLLOWER
I20260924 05:08:58.215596 2619 consensus_queue.cc:260] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.215808 2623 tablet_bootstrap.cc:654] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:58.215875 2619 ts_tablet_manager.cc:1428] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:08:58.215952 2598 heartbeater.cc:503] Master 127.0.98.62:33257 was elected leader, sending a full tablet report...
I20260924 05:08:58.216117 2623 log.cc:824] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:08:58.216847 2623 tablet_bootstrap.cc:492] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: No bootstrap required, opened a new log
I20260924 05:08:58.216945 2623 ts_tablet_manager.cc:1397] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:08:58.218412 2623 raft_consensus.cc:348] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.218547 2623 raft_consensus.cc:374] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:58.218590 2623 raft_consensus.cc:729] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa43c6af00ed46b49b1794ca3484a2fb, State: Initialized, Role: FOLLOWER
I20260924 05:08:58.218698 2623 consensus_queue.cc:260] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.218987 2623 ts_tablet_manager.cc:1428] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:08:58.218994 2458 heartbeater.cc:503] Master 127.0.98.62:33257 was elected leader, sending a full tablet report...
I20260924 05:08:58.230842 2624 raft_consensus.cc:482] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:08:58.230957 2624 raft_consensus.cc:504] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.231248 2624 leader_election.cc:288] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043), fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389)
I20260924 05:08:58.234081 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.231566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59325 (local address 127.0.98.2:46389)
0924 05:08:58.231599 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.231601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.231697 (+ 96us) server_negotiation.cc:407] Connection header received
0924 05:08:58.231750 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.231751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.231761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.231814 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:58.231952 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.232406 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.233252 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.233729 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.233814 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.233823 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.233971 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.234013 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:08:58.234099 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.231481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47973 (local address 127.0.98.1:33043)
0924 05:08:58.231526 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.231528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.231537 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:08:58.231590 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.231591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.231603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.231664 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:58.231819 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.232299 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.233189 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.233635 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.233701 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.233711 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.233959 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.234012 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:08:58.234113 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.231430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.98.3:47973)
0924 05:08:58.231480 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.231491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.231530 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.231669 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.231670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.231672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:58.231731 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.231733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.232387 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.232387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.233102 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.233104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.233709 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.233710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.233849 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.233853 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.233943 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.234009 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:08:58.234429 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.231429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.98.3:59325)
0924 05:08:58.231631 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.231638 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.231702 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.231817 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.231818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.231819 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:58.231879 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.231880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.232486 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.232487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.233167 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.233168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.233814 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.233814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.233880 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.233884 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.234324 (+ 440us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.234360 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":56,"thread_start_us":90,"threads_started":1}
I20260924 05:08:58.234848 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c71e70b6ed47809576b89ef106359d" candidate_uuid: "a4353208709943d985c0f69eedf745fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" is_pre_election: true
I20260924 05:08:58.234897 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c71e70b6ed47809576b89ef106359d" candidate_uuid: "a4353208709943d985c0f69eedf745fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" is_pre_election: true
I20260924 05:08:58.235010 2268 raft_consensus.cc:2454] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4353208709943d985c0f69eedf745fe in term 0.
I20260924 05:08:58.235041 2412 raft_consensus.cc:2454] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4353208709943d985c0f69eedf745fe in term 0.
I20260924 05:08:58.235278 2483 leader_election.cc:302] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4353208709943d985c0f69eedf745fe, fa43c6af00ed46b49b1794ca3484a2fb; no voters:
I20260924 05:08:58.235423 2624 raft_consensus.cc:2781] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:08:58.235471 2624 raft_consensus.cc:482] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:08:58.235508 2624 raft_consensus.cc:3037] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:58.236478 2624 raft_consensus.cc:504] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.236733 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c71e70b6ed47809576b89ef106359d" candidate_uuid: "a4353208709943d985c0f69eedf745fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2"
I20260924 05:08:58.236759 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23c71e70b6ed47809576b89ef106359d" candidate_uuid: "a4353208709943d985c0f69eedf745fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa43c6af00ed46b49b1794ca3484a2fb"
I20260924 05:08:58.236814 2268 raft_consensus.cc:3037] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:58.236828 2412 raft_consensus.cc:3037] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:58.237739 2412 raft_consensus.cc:2454] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4353208709943d985c0f69eedf745fe in term 1.
I20260924 05:08:58.237812 2268 raft_consensus.cc:2454] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4353208709943d985c0f69eedf745fe in term 1.
I20260924 05:08:58.237913 2483 leader_election.cc:302] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4353208709943d985c0f69eedf745fe, fa43c6af00ed46b49b1794ca3484a2fb; no voters:
I20260924 05:08:58.238103 2624 leader_election.cc:288] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [CANDIDATE]: Term 1 election: Requested vote from peers e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043), fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389)
I20260924 05:08:58.238142 2628 raft_consensus.cc:2781] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:58.238240 2628 raft_consensus.cc:686] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 1 LEADER]: Becoming Leader. State: Replica: a4353208709943d985c0f69eedf745fe, State: Running, Role: LEADER
I20260924 05:08:58.238317 2628 consensus_queue.cc:237] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.239183 2115 catalog_manager.cc:5495] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe reported cstate change: term changed from 0 to 1, leader changed from <none> to a4353208709943d985c0f69eedf745fe (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a4353208709943d985c0f69eedf745fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } health_report { overall_health: HEALTHY } } }
Sep 24 05:08:58 dist-test-slave-mr2f krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:58.269747 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.263496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.0.98.62:33257)
0924 05:08:58.263571 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.263573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.263727 (+ 154us) server_negotiation.cc:407] Connection header received
0924 05:08:58.263764 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.263765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.263773 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.263822 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:58.264220 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.264689 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.265357 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.265525 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.265571 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.267543 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:58.267544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:58.267544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:58.267550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:58.268816 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.269143 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.269144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.269144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.269189 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:58.269354 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:58.269354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:58.269355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:58.269453 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:58.269501 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.269623 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.269685 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 05:08:58.269863 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.263452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33257 (local address 127.0.0.1:45158)
0924 05:08:58.263639 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.263649 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.263725 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.263840 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.263841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.264117 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:08:58.264159 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.264161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.264754 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.264755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.265294 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.265295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.265584 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.265585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.265622 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.266151 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:08:58.266155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:08:58.267472 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:08:58.268887 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.268892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.268894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.269079 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.269245 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:08:58.269246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:08:58.269246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:08:58.269302 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:08:58.269506 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:08:58.269512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:08:58.269551 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.269607 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.269750 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":57,"thread_start_us":115,"threads_started":1}
I20260924 05:08:58.271865 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45158:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 05:08:58.272042 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:08:58.273397 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 81e4b8227d784cb58cf0e8e893543841 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e537b29ec73440adb45207e4ac2fe3a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:08:58.273397 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 81e4b8227d784cb58cf0e8e893543841 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e537b29ec73440adb45207e4ac2fe3a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:08:58.273582 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e4b8227d784cb58cf0e8e893543841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:58.273587 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e4b8227d784cb58cf0e8e893543841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:58.273638 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 81e4b8227d784cb58cf0e8e893543841 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e537b29ec73440adb45207e4ac2fe3a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:08:58.273802 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e4b8227d784cb58cf0e8e893543841. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 05:08:58.274854 2459 tablet.cc:2369] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:08:58.275199 2618 tablet_bootstrap.cc:492] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: Bootstrap starting.
I20260924 05:08:58.275205 2623 tablet_bootstrap.cc:492] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: Bootstrap starting.
I20260924 05:08:58.275233 2619 tablet_bootstrap.cc:492] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: Bootstrap starting.
I20260924 05:08:58.275640 2619 tablet_bootstrap.cc:654] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:58.275956 2618 tablet_bootstrap.cc:654] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:58.275960 2623 tablet_bootstrap.cc:654] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:58.276799 2618 tablet_bootstrap.cc:492] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: No bootstrap required, opened a new log
I20260924 05:08:58.276853 2618 ts_tablet_manager.cc:1397] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:08:58.276885 2623 tablet_bootstrap.cc:492] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: No bootstrap required, opened a new log
I20260924 05:08:58.276890 2619 tablet_bootstrap.cc:492] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: No bootstrap required, opened a new log
I20260924 05:08:58.276929 2623 ts_tablet_manager.cc:1397] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:08:58.276933 2619 ts_tablet_manager.cc:1397] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:08:58.277024 2618 raft_consensus.cc:348] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.277067 2623 raft_consensus.cc:348] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.277112 2618 raft_consensus.cc:374] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:58.277130 2623 raft_consensus.cc:374] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:58.277136 2618 raft_consensus.cc:729] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fcae7ed09d4fa6986255a87a28d5f2, State: Initialized, Role: FOLLOWER
I20260924 05:08:58.277092 2619 raft_consensus.cc:348] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.277159 2623 raft_consensus.cc:729] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa43c6af00ed46b49b1794ca3484a2fb, State: Initialized, Role: FOLLOWER
I20260924 05:08:58.277168 2619 raft_consensus.cc:374] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:58.277189 2619 raft_consensus.cc:729] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4353208709943d985c0f69eedf745fe, State: Initialized, Role: FOLLOWER
I20260924 05:08:58.277184 2618 consensus_queue.cc:260] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.277199 2623 consensus_queue.cc:260] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.277235 2619 consensus_queue.cc:260] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.277272 2623 ts_tablet_manager.cc:1428] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:58.277271 2618 ts_tablet_manager.cc:1428] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:58.277328 2619 ts_tablet_manager.cc:1428] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:58.277380 2622 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.277459 2622 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:58.277588 2626 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.277666 2626 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:58.277791 2628 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.277860 2628 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:58.307982 2622 raft_consensus.cc:482] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:08:58.308071 2622 raft_consensus.cc:504] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.308409 2622 leader_election.cc:288] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389), a4353208709943d985c0f69eedf745fe (127.0.98.3:46861)
I20260924 05:08:58.311335 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.308579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.98.1:45147)
0924 05:08:58.308648 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.308660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.308714 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.308902 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.308903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.308905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:58.308985 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.308987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.309688 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.309689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.310389 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.310390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.311084 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.311085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.311162 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.311166 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.311226 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.311271 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 05:08:58.311357 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.308667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45147 (local address 127.0.98.2:46389)
0924 05:08:58.308746 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.308749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.308759 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:58.308800 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.308801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.308810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.308872 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:58.309057 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.309604 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.310495 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.311019 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.311084 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.311093 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.311251 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.311289 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 05:08:58.311610 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.308717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46861 (local address 127.0.98.1:58285)
0924 05:08:58.308889 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.308898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.308968 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.309124 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.309125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.309127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:08:58.309207 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.309208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.309813 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.309814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.310500 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.310502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.311274 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.311275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.311338 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.311341 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.311400 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.311529 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":130,"threads_started":1}
I20260924 05:08:58.311596 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4b8227d784cb58cf0e8e893543841" candidate_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" is_pre_election: true
I20260924 05:08:58.311734 2412 raft_consensus.cc:2454] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1fcae7ed09d4fa6986255a87a28d5f2 in term 0.
I20260924 05:08:58.311923 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.308833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58285 (local address 127.0.98.3:46861)
0924 05:08:58.308889 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.308891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.308941 (+ 50us) server_negotiation.cc:407] Connection header received
0924 05:08:58.309027 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.309028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.309039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.309088 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:08:58.309264 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.309716 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.310597 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.311174 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.311238 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.311247 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.311813 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.311861 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:08:58.311920 2203 leader_election.cc:302] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1fcae7ed09d4fa6986255a87a28d5f2, fa43c6af00ed46b49b1794ca3484a2fb; no voters:
I20260924 05:08:58.312059 2622 raft_consensus.cc:2781] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:08:58.312105 2622 raft_consensus.cc:482] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:08:58.312127 2622 raft_consensus.cc:3037] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:58.312197 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4b8227d784cb58cf0e8e893543841" candidate_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4353208709943d985c0f69eedf745fe" is_pre_election: true
I20260924 05:08:58.312327 2552 raft_consensus.cc:2454] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1fcae7ed09d4fa6986255a87a28d5f2 in term 0.
I20260924 05:08:58.312915 2622 raft_consensus.cc:504] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.313021 2622 leader_election.cc:288] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [CANDIDATE]: Term 1 election: Requested vote from peers fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389), a4353208709943d985c0f69eedf745fe (127.0.98.3:46861)
I20260924 05:08:58.313230 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4b8227d784cb58cf0e8e893543841" candidate_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa43c6af00ed46b49b1794ca3484a2fb"
I20260924 05:08:58.313248 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e4b8227d784cb58cf0e8e893543841" candidate_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4353208709943d985c0f69eedf745fe"
I20260924 05:08:58.313344 2552 raft_consensus.cc:3037] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:58.313347 2412 raft_consensus.cc:3037] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:58.313958 2412 raft_consensus.cc:2454] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1fcae7ed09d4fa6986255a87a28d5f2 in term 1.
I20260924 05:08:58.313968 2552 raft_consensus.cc:2454] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1fcae7ed09d4fa6986255a87a28d5f2 in term 1.
I20260924 05:08:58.314102 2203 leader_election.cc:302] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4353208709943d985c0f69eedf745fe, e1fcae7ed09d4fa6986255a87a28d5f2; no voters:
I20260924 05:08:58.314181 2622 raft_consensus.cc:2781] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:58.314333 2622 raft_consensus.cc:686] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 LEADER]: Becoming Leader. State: Replica: e1fcae7ed09d4fa6986255a87a28d5f2, State: Running, Role: LEADER
I20260924 05:08:58.314455 2622 consensus_queue.cc:237] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.314960 2640 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: TxnStatusTablet state changed. Reason: New leader e1fcae7ed09d4fa6986255a87a28d5f2. Latest consensus state: current_term: 1 leader_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.315047 2640 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:08:58.315280 2642 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 05:08:58.315315 2115 catalog_manager.cc:5495] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } health_report { overall_health: UNKNOWN } } }
I20260924 05:08:58.315414 2642 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 05:08:58.325107 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.322654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.0.98.1:33043)
0924 05:08:58.322712 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.322714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.322740 (+ 26us) server_negotiation.cc:407] Connection header received
0924 05:08:58.322840 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.322841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.322857 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.322921 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:58.323057 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.323598 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.324340 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.324545 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.324627 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.324801 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.324901 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.324988 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.325031 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.325050 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:08:58.325165 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.322567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.0.1:46370)
0924 05:08:58.322634 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.322644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.322708 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.322925 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.322926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.322926 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:08:58.322984 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.322986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.323678 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.323678 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.324239 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.324241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.324630 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.324631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.324701 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.324713 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.324968 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.324976 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.325025 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.325073 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:08:58.326648 2412 raft_consensus.cc:1261] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Refusing update from remote peer e1fcae7ed09d4fa6986255a87a28d5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:58.326673 2552 raft_consensus.cc:1261] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Refusing update from remote peer e1fcae7ed09d4fa6986255a87a28d5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:58.326917 2622 consensus_queue.cc:1035] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:58.327005 2640 consensus_queue.cc:1035] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:58.328076 2628 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: TxnStatusTablet state changed. Reason: New leader e1fcae7ed09d4fa6986255a87a28d5f2. Latest consensus state: current_term: 1 leader_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.328099 2626 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: TxnStatusTablet state changed. Reason: New leader e1fcae7ed09d4fa6986255a87a28d5f2. Latest consensus state: current_term: 1 leader_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.328168 2626 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:58.328169 2628 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:58.329349 2622 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.329408 2640 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.329486 2640 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:08:58.329588 2626 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.329664 2626 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P fa43c6af00ed46b49b1794ca3484a2fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:58.329424 2622 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P e1fcae7ed09d4fa6986255a87a28d5f2: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:08:58.329936 2646 mvcc.cc:205] Tried to move back new op lower bound from 7332767900983693312 to 7332767900936552448. Current Snapshot: MvccSnapshot[applied={T|T < 7332767900983693312}]
I20260924 05:08:58.330101 2645 mvcc.cc:205] Tried to move back new op lower bound from 7332767900983693312 to 7332767900936552448. Current Snapshot: MvccSnapshot[applied={T|T < 7332767900983693312}]
I20260924 05:08:58.330487 2628 tablet_replica.cc:439] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } } }
I20260924 05:08:58.330565 2628 tablet_replica.cc:442] T 81e4b8227d784cb58cf0e8e893543841 P a4353208709943d985c0f69eedf745fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:08:58.335031 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.332748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52404 (local address 127.0.98.3:46861)
0924 05:08:58.332804 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.332806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.332831 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:08:58.332855 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.332856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.332869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.332930 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:58.333031 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.333464 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.334224 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.334431 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.334519 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.334680 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.334819 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.334893 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.334941 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.334962 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:08:58.339236 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.337215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60847 (local address 127.0.98.1:33043)
0924 05:08:58.337248 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.337250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.337260 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:58.337321 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.337322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.337330 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.337382 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:58.337481 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.337896 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.338580 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.338798 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.338858 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.339037 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.339064 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.339113 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.339167 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.339181 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:08:58.339262 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.337154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.98.3:60847)
0924 05:08:58.337221 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.337231 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.337274 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.337384 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.337385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.337385 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:08:58.337436 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.337437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.337973 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.337974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.338502 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.338504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.338860 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.338861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.338920 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.338936 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.339116 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.339118 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.339163 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.339200 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 05:08:58.341704 2268 raft_consensus.cc:1261] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Refusing update from remote peer a4353208709943d985c0f69eedf745fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:58.341938 2624 consensus_queue.cc:1035] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:58.341987 2412 raft_consensus.cc:1261] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Refusing update from remote peer a4353208709943d985c0f69eedf745fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:58.342154 2624 consensus_queue.cc:1035] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:58.343086 2647 mvcc.cc:205] Tried to move back new op lower bound from 7332767901045743616 to 7332767900624891904. Current Snapshot: MvccSnapshot[applied={T|T < 7332767901045743616}]
I20260924 05:08:58.350165 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.347698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46861 (local address 127.0.98.1:56875)
0924 05:08:58.347773 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.347784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.347823 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.348052 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.348053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.348054 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:08:58.348120 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.348121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.348736 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.348737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.349353 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.349356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.349764 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.349765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.349826 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.349846 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.350022 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.350024 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.350062 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.350104 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 05:08:58.350162 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.347833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56875 (local address 127.0.98.3:46861)
0924 05:08:58.347959 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.347960 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.347968 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:08:58.347987 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.347988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.347998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.348050 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:58.348210 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.348658 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.349436 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.349687 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.349762 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.349921 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.349951 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.349998 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.350084 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.350100 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260924 05:08:58.360361 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:08:58.360460 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:08:58.361110 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf: Sending DeleteTablet for 3 replicas of tablet 23c71e70b6ed47809576b89ef106359d
I20260924 05:08:58.361512 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c71e70b6ed47809576b89ef106359d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35392
I20260924 05:08:58.361728 2664 tablet_replica.cc:330] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: stopping tablet replica
I20260924 05:08:58.361711 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c71e70b6ed47809576b89ef106359d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52402
I20260924 05:08:58.362310 2664 raft_consensus.cc:2229] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:08:58.362387 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 23c71e70b6ed47809576b89ef106359d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260924 05:08:58.362418 2664 raft_consensus.cc:2258] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:58.362567 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:08:58.362704 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:08:58.363125 2664 ts_tablet_manager.cc:1905] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:58.364064 2392 tablet_service.cc:1467] Processing CreateTablet for tablet b2e2c6ab4fb842ef93e052c411b5dedf (DEFAULT_TABLE table=test-table [id=ba895dc92f14448aa2e4f4433a8be7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:58.364177 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e2c6ab4fb842ef93e052c411b5dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:58.365329 2623 tablet_bootstrap.cc:492] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: Bootstrap starting.
I20260924 05:08:58.365499 2532 tablet_service.cc:1467] Processing CreateTablet for tablet b2e2c6ab4fb842ef93e052c411b5dedf (DEFAULT_TABLE table=test-table [id=ba895dc92f14448aa2e4f4433a8be7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:58.365597 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e2c6ab4fb842ef93e052c411b5dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:58.365711 2623 tablet_bootstrap.cc:654] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:58.366400 2664 ts_tablet_manager.cc:1918] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:08:58.366523 2664 log.cc:1196] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/23c71e70b6ed47809576b89ef106359d
I20260924 05:08:58.366902 2252 tablet_service.cc:1467] Processing CreateTablet for tablet b2e2c6ab4fb842ef93e052c411b5dedf (DEFAULT_TABLE table=test-table [id=ba895dc92f14448aa2e4f4433a8be7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:58.367015 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e2c6ab4fb842ef93e052c411b5dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:58.366856 2664 ts_tablet_manager.cc:1939] T 23c71e70b6ed47809576b89ef106359d P fa43c6af00ed46b49b1794ca3484a2fb: Deleting consensus metadata
I20260924 05:08:58.367693 2099 catalog_manager.cc:4841] TS fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389): tablet 23c71e70b6ed47809576b89ef106359d (table test-table [id=3a9b962dc5a146f9a008479de23bc13c]) successfully deleted
I20260924 05:08:58.368080 2623 tablet_bootstrap.cc:492] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: No bootstrap required, opened a new log
I20260924 05:08:58.368121 2623 ts_tablet_manager.cc:1397] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 05:08:58.368247 2623 raft_consensus.cc:348] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.368331 2623 raft_consensus.cc:374] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:58.368352 2623 raft_consensus.cc:729] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa43c6af00ed46b49b1794ca3484a2fb, State: Initialized, Role: FOLLOWER
I20260924 05:08:58.368407 2618 tablet_bootstrap.cc:492] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: Bootstrap starting.
I20260924 05:08:58.368394 2623 consensus_queue.cc:260] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.368523 2623 ts_tablet_manager.cc:1428] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:58.368741 2619 tablet_bootstrap.cc:492] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: Bootstrap starting.
I20260924 05:08:58.368799 2618 tablet_bootstrap.cc:654] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:58.369056 2666 tablet_replica.cc:330] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: stopping tablet replica
I20260924 05:08:58.369165 2666 raft_consensus.cc:2229] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:08:58.369218 2666 raft_consensus.cc:2258] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:58.369170 2619 tablet_bootstrap.cc:654] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:58.368878 2665 tablet_replica.cc:330] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: stopping tablet replica
I20260924 05:08:58.369691 2665 raft_consensus.cc:2229] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:08:58.369789 2665 raft_consensus.cc:2258] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:58.370258 2666 ts_tablet_manager.cc:1905] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:58.371006 2665 ts_tablet_manager.cc:1905] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:58.371698 2666 ts_tablet_manager.cc:1918] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:08:58.371770 2666 log.cc:1196] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/23c71e70b6ed47809576b89ef106359d
I20260924 05:08:58.372037 2666 ts_tablet_manager.cc:1939] T 23c71e70b6ed47809576b89ef106359d P e1fcae7ed09d4fa6986255a87a28d5f2: Deleting consensus metadata
I20260924 05:08:58.372462 2100 catalog_manager.cc:4841] TS e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043): tablet 23c71e70b6ed47809576b89ef106359d (table test-table [id=3a9b962dc5a146f9a008479de23bc13c]) successfully deleted
I20260924 05:08:58.372982 2665 ts_tablet_manager.cc:1918] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:08:58.373067 2665 log.cc:1196] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/23c71e70b6ed47809576b89ef106359d
I20260924 05:08:58.373353 2665 ts_tablet_manager.cc:1939] T 23c71e70b6ed47809576b89ef106359d P a4353208709943d985c0f69eedf745fe: Deleting consensus metadata
I20260924 05:08:58.373711 2099 catalog_manager.cc:4841] TS a4353208709943d985c0f69eedf745fe (127.0.98.3:46861): tablet 23c71e70b6ed47809576b89ef106359d (table test-table [id=3a9b962dc5a146f9a008479de23bc13c]) successfully deleted
I20260924 05:08:58.373948 2619 tablet_bootstrap.cc:492] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: No bootstrap required, opened a new log
I20260924 05:08:58.373996 2619 ts_tablet_manager.cc:1397] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 05:08:58.374131 2619 raft_consensus.cc:348] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.374195 2619 raft_consensus.cc:374] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:58.374217 2619 raft_consensus.cc:729] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4353208709943d985c0f69eedf745fe, State: Initialized, Role: FOLLOWER
I20260924 05:08:58.374265 2619 consensus_queue.cc:260] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.374361 2619 ts_tablet_manager.cc:1428] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:58.379096 2618 tablet_bootstrap.cc:492] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: No bootstrap required, opened a new log
I20260924 05:08:58.379161 2618 ts_tablet_manager.cc:1397] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260924 05:08:58.379302 2618 raft_consensus.cc:348] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.379372 2618 raft_consensus.cc:374] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:58.379400 2618 raft_consensus.cc:729] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fcae7ed09d4fa6986255a87a28d5f2, State: Initialized, Role: FOLLOWER
I20260924 05:08:58.379437 2618 consensus_queue.cc:260] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.379550 2618 ts_tablet_manager.cc:1428] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:58.387871 2640 raft_consensus.cc:482] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:08:58.387938 2640 raft_consensus.cc:504] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.388067 2640 leader_election.cc:288] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389), a4353208709943d985c0f69eedf745fe (127.0.98.3:46861)
I20260924 05:08:58.388258 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e2c6ab4fb842ef93e052c411b5dedf" candidate_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4353208709943d985c0f69eedf745fe" is_pre_election: true
I20260924 05:08:58.388259 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e2c6ab4fb842ef93e052c411b5dedf" candidate_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" is_pre_election: true
I20260924 05:08:58.388332 2412 raft_consensus.cc:2454] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1fcae7ed09d4fa6986255a87a28d5f2 in term 0.
I20260924 05:08:58.388332 2552 raft_consensus.cc:2454] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1fcae7ed09d4fa6986255a87a28d5f2 in term 0.
I20260924 05:08:58.388474 2203 leader_election.cc:302] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4353208709943d985c0f69eedf745fe, e1fcae7ed09d4fa6986255a87a28d5f2; no voters:
I20260924 05:08:58.388552 2640 raft_consensus.cc:2781] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:08:58.388582 2640 raft_consensus.cc:482] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:08:58.388603 2640 raft_consensus.cc:3037] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:58.389262 2640 raft_consensus.cc:504] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.389397 2640 leader_election.cc:288] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [CANDIDATE]: Term 1 election: Requested vote from peers fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389), a4353208709943d985c0f69eedf745fe (127.0.98.3:46861)
I20260924 05:08:58.389609 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e2c6ab4fb842ef93e052c411b5dedf" candidate_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa43c6af00ed46b49b1794ca3484a2fb"
I20260924 05:08:58.389622 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e2c6ab4fb842ef93e052c411b5dedf" candidate_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4353208709943d985c0f69eedf745fe"
I20260924 05:08:58.389678 2552 raft_consensus.cc:3037] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:58.389678 2412 raft_consensus.cc:3037] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:08:58.390375 2552 raft_consensus.cc:2454] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1fcae7ed09d4fa6986255a87a28d5f2 in term 1.
I20260924 05:08:58.390413 2412 raft_consensus.cc:2454] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1fcae7ed09d4fa6986255a87a28d5f2 in term 1.
I20260924 05:08:58.390563 2203 leader_election.cc:302] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1fcae7ed09d4fa6986255a87a28d5f2, fa43c6af00ed46b49b1794ca3484a2fb; no voters:
I20260924 05:08:58.390674 2640 raft_consensus.cc:2781] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:08:58.390753 2640 raft_consensus.cc:686] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 LEADER]: Becoming Leader. State: Replica: e1fcae7ed09d4fa6986255a87a28d5f2, State: Running, Role: LEADER
I20260924 05:08:58.390818 2640 consensus_queue.cc:237] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:58.391276 2115 catalog_manager.cc:5495] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } health_report { overall_health: UNKNOWN } } }
I20260924 05:08:58.404528 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.402244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.0.98.1:33043)
0924 05:08:58.402280 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.402283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.402315 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:08:58.402382 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.402383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.402393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.402436 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:58.402543 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.403099 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.403840 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.404034 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.404112 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.404261 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.404299 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.404356 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.404416 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.404432 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:08:58.408815 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.406375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40009 (local address 127.0.98.1:33043)
0924 05:08:58.406402 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:58.406404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:58.406438 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:08:58.406496 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:58.406497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:58.406506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:58.406558 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:08:58.406658 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.407402 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.408178 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.408375 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.408440 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.408576 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.408601 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.408660 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:58.408715 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 05:08:58.408729 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":15}
I20260924 05:08:58.408815 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:58.406349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.98.1:40009)
0924 05:08:58.406398 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:08:58.406409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:08:58.406451 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:08:58.406561 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:08:58.406562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:08:58.406563 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:08:58.406595 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.406596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.407505 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.407506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.408079 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:08:58.408081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:08:58.408447 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:08:58.408448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:08:58.408493 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:58.408496 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:08:58.408660 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:08:58.408661 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:08:58.408711 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:08:58.408744 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:08:58.411105 2412 raft_consensus.cc:1261] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Refusing update from remote peer e1fcae7ed09d4fa6986255a87a28d5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:58.411095 2552 raft_consensus.cc:1261] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Refusing update from remote peer e1fcae7ed09d4fa6986255a87a28d5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:08:58.411283 2651 consensus_queue.cc:1035] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:08:58.411379 2622 consensus_queue.cc:1035] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 05:08:58.417155 2645 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 05:08:59.937422 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:08:59.937510 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:08:59.938261 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf: Sending DeleteTablet for 3 replicas of tablet b2e2c6ab4fb842ef93e052c411b5dedf
I20260924 05:08:59.938510 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e2c6ab4fb842ef93e052c411b5dedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260924 05:08:59.938556 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e2c6ab4fb842ef93e052c411b5dedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52402
I20260924 05:08:59.938565 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e2c6ab4fb842ef93e052c411b5dedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:08:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35392
I20260924 05:08:59.938690 2693 tablet_replica.cc:330] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: stopping tablet replica
I20260924 05:08:59.938738 2693 raft_consensus.cc:2229] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:08:59.938772 2694 tablet_replica.cc:330] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: stopping tablet replica
I20260924 05:08:59.938823 2694 raft_consensus.cc:2229] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:08:59.938854 2693 raft_consensus.cc:2258] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:59.938877 2694 raft_consensus.cc:2258] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:59.938970 2695 tablet_replica.cc:330] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: stopping tablet replica
I20260924 05:08:59.939023 2695 raft_consensus.cc:2229] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:08:59.939055 2695 raft_consensus.cc:2258] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:08:59.939260 2695 ts_tablet_manager.cc:1905] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:59.939268 2693 ts_tablet_manager.cc:1905] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:59.940836 2693 ts_tablet_manager.cc:1918] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:08:59.940903 2693 log.cc:1196] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/b2e2c6ab4fb842ef93e052c411b5dedf
I20260924 05:08:59.941154 2693 ts_tablet_manager.cc:1939] T b2e2c6ab4fb842ef93e052c411b5dedf P e1fcae7ed09d4fa6986255a87a28d5f2: Deleting consensus metadata
I20260924 05:08:59.941155 2695 ts_tablet_manager.cc:1918] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:08:59.941210 2695 log.cc:1196] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/b2e2c6ab4fb842ef93e052c411b5dedf
I20260924 05:08:59.941430 2695 ts_tablet_manager.cc:1939] T b2e2c6ab4fb842ef93e052c411b5dedf P fa43c6af00ed46b49b1794ca3484a2fb: Deleting consensus metadata
I20260924 05:08:59.941447 2100 catalog_manager.cc:4841] TS e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043): tablet b2e2c6ab4fb842ef93e052c411b5dedf (table test-table [id=ba895dc92f14448aa2e4f4433a8be7b9]) successfully deleted
I20260924 05:08:59.941566 2694 ts_tablet_manager.cc:1905] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:08:59.941798 2099 catalog_manager.cc:4841] TS fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389): tablet b2e2c6ab4fb842ef93e052c411b5dedf (table test-table [id=ba895dc92f14448aa2e4f4433a8be7b9]) successfully deleted
I20260924 05:08:59.943397 2694 ts_tablet_manager.cc:1918] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:08:59.943462 2694 log.cc:1196] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/b2e2c6ab4fb842ef93e052c411b5dedf
I20260924 05:08:59.943686 2694 ts_tablet_manager.cc:1939] T b2e2c6ab4fb842ef93e052c411b5dedf P a4353208709943d985c0f69eedf745fe: Deleting consensus metadata
I20260924 05:08:59.944020 2099 catalog_manager.cc:4841] TS a4353208709943d985c0f69eedf745fe (127.0.98.3:46861): tablet b2e2c6ab4fb842ef93e052c411b5dedf (table test-table [id=ba895dc92f14448aa2e4f4433a8be7b9]) successfully deleted
Sep 24 05:08:59 dist-test-slave-mr2f krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226539, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:08:59 dist-test-slave-mr2f krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226539, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:08:59.968582 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:08:59.955567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.0.98.62:33257)
0924 05:08:59.955686 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 05:08:59.955688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:08:59.955698 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:08:59.955760 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:08:59.955761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:08:59.955770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:08:59.955828 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:08:59.956334 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:59.956825 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:59.957490 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:08:59.957673 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:08:59.957720 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:08:59.966138 (+ 8418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:08:59.966139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:08:59.966140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:08:59.966146 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:08:59.967646 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:59.967978 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:59.967979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:59.967979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:59.968029 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:08:59.968173 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:08:59.968174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:08:59.968174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:08:59.968260 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:08:59.968324 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:08:59.968411 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:08:59.968468 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260924 05:08:59.970027 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:08:59.970193 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:08:59.971671 2392 tablet_service.cc:1467] Processing CreateTablet for tablet aecabe536c2d4c568290b39df6e5a3d7 (DEFAULT_TABLE table=test-table [id=a80db279d7244cd79240c6c1037ebaa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:59.971676 2532 tablet_service.cc:1467] Processing CreateTablet for tablet aecabe536c2d4c568290b39df6e5a3d7 (DEFAULT_TABLE table=test-table [id=a80db279d7244cd79240c6c1037ebaa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:59.971812 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecabe536c2d4c568290b39df6e5a3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:59.971818 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecabe536c2d4c568290b39df6e5a3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:59.972791 2705 tablet_bootstrap.cc:492] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: Bootstrap starting.
I20260924 05:08:59.972844 2252 tablet_service.cc:1467] Processing CreateTablet for tablet aecabe536c2d4c568290b39df6e5a3d7 (DEFAULT_TABLE table=test-table [id=a80db279d7244cd79240c6c1037ebaa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:08:59.972941 2704 tablet_bootstrap.cc:492] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: Bootstrap starting.
I20260924 05:08:59.972962 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecabe536c2d4c568290b39df6e5a3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:08:59.973158 2705 tablet_bootstrap.cc:654] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:59.973351 2704 tablet_bootstrap.cc:654] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:59.973802 2705 tablet_bootstrap.cc:492] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: No bootstrap required, opened a new log
I20260924 05:08:59.973863 2705 ts_tablet_manager.cc:1397] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:08:59.973999 2705 raft_consensus.cc:348] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:59.974069 2705 raft_consensus.cc:374] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:59.974089 2705 raft_consensus.cc:729] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa43c6af00ed46b49b1794ca3484a2fb, State: Initialized, Role: FOLLOWER
I20260924 05:08:59.974154 2705 consensus_queue.cc:260] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:59.974331 2705 ts_tablet_manager.cc:1428] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:08:59.974502 2709 tablet_bootstrap.cc:492] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: Bootstrap starting.
I20260924 05:08:59.974896 2704 tablet_bootstrap.cc:492] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: No bootstrap required, opened a new log
I20260924 05:08:59.974948 2704 ts_tablet_manager.cc:1397] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:08:59.975025 2709 tablet_bootstrap.cc:654] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: Neither blocks nor log segments found. Creating new log.
I20260924 05:08:59.975075 2704 raft_consensus.cc:348] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:59.975150 2704 raft_consensus.cc:374] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:59.975173 2704 raft_consensus.cc:729] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4353208709943d985c0f69eedf745fe, State: Initialized, Role: FOLLOWER
I20260924 05:08:59.975226 2704 consensus_queue.cc:260] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:59.975410 2704 ts_tablet_manager.cc:1428] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:08:59.975747 2709 tablet_bootstrap.cc:492] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: No bootstrap required, opened a new log
I20260924 05:08:59.975809 2709 ts_tablet_manager.cc:1397] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:08:59.975914 2709 raft_consensus.cc:348] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:59.975982 2709 raft_consensus.cc:374] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:08:59.976012 2709 raft_consensus.cc:729] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fcae7ed09d4fa6986255a87a28d5f2, State: Initialized, Role: FOLLOWER
I20260924 05:08:59.976065 2709 consensus_queue.cc:260] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:08:59.976208 2709 ts_tablet_manager.cc:1428] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:00.098663 2708 raft_consensus.cc:482] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:00.098779 2708 raft_consensus.cc:504] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.099172 2708 leader_election.cc:288] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043), a4353208709943d985c0f69eedf745fe (127.0.98.3:46861)
I20260924 05:09:00.102138 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:00.099389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49589 (local address 127.0.98.1:33043)
0924 05:09:00.099564 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:00.099566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:00.099578 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:09:00.099644 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:00.099645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:00.099656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:00.099725 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:00.099898 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.100404 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.101390 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.101772 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.101832 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:00.101841 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:00.102016 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 05:09:00.102056 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
I20260924 05:09:00.102185 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:00.099374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46861 (local address 127.0.98.2:51851)
0924 05:09:00.099547 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:00.099558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:00.099614 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:00.099809 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:00.099810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:00.099813 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:00.099882 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:00.099884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.100490 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.100491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:00.101309 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:00.101333 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.101902 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.101902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:00.101953 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:00.101956 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:00.102028 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:09:00.102064 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":50,"thread_start_us":107,"threads_started":1}
I20260924 05:09:00.102175 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:00.099438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51851 (local address 127.0.98.3:46861)
0924 05:09:00.099611 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:00.099613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:00.099623 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:00.099676 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:00.099676 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:00.099693 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:00.099781 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:00.099963 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.100407 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.101450 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.101795 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.101883 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:00.101895 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:00.102045 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 05:09:00.102086 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260924 05:09:00.102200 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:00.099329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.98.2:49589)
0924 05:09:00.099509 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:00.099522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:00.099581 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:00.099752 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:00.099753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:00.099756 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:00.099817 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:00.099819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.100491 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.100492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:00.101301 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:00.101303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.101853 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.101854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:00.101934 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:00.101937 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:00.101989 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:09:00.102046 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":31,"thread_start_us":108,"threads_started":1}
I20260924 05:09:00.102406 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecabe536c2d4c568290b39df6e5a3d7" candidate_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4353208709943d985c0f69eedf745fe" is_pre_election: true
I20260924 05:09:00.102519 2552 raft_consensus.cc:2454] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa43c6af00ed46b49b1794ca3484a2fb in term 0.
I20260924 05:09:00.102587 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecabe536c2d4c568290b39df6e5a3d7" candidate_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" is_pre_election: true
I20260924 05:09:00.102663 2268 raft_consensus.cc:2454] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa43c6af00ed46b49b1794ca3484a2fb in term 0.
I20260924 05:09:00.102699 2343 leader_election.cc:302] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4353208709943d985c0f69eedf745fe, fa43c6af00ed46b49b1794ca3484a2fb; no voters:
I20260924 05:09:00.102864 2708 raft_consensus.cc:2781] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:00.102908 2708 raft_consensus.cc:482] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:00.102922 2708 raft_consensus.cc:3037] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:00.103628 2708 raft_consensus.cc:504] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.103758 2708 leader_election.cc:288] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [CANDIDATE]: Term 1 election: Requested vote from peers e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043), a4353208709943d985c0f69eedf745fe (127.0.98.3:46861)
I20260924 05:09:00.103907 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecabe536c2d4c568290b39df6e5a3d7" candidate_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4353208709943d985c0f69eedf745fe"
I20260924 05:09:00.103906 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecabe536c2d4c568290b39df6e5a3d7" candidate_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2"
I20260924 05:09:00.103974 2268 raft_consensus.cc:3037] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:00.103976 2552 raft_consensus.cc:3037] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:00.104657 2268 raft_consensus.cc:2454] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa43c6af00ed46b49b1794ca3484a2fb in term 1.
I20260924 05:09:00.104656 2552 raft_consensus.cc:2454] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa43c6af00ed46b49b1794ca3484a2fb in term 1.
I20260924 05:09:00.104813 2343 leader_election.cc:302] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4353208709943d985c0f69eedf745fe, fa43c6af00ed46b49b1794ca3484a2fb; no voters:
I20260924 05:09:00.104904 2708 raft_consensus.cc:2781] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:00.105057 2708 raft_consensus.cc:686] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 LEADER]: Becoming Leader. State: Replica: fa43c6af00ed46b49b1794ca3484a2fb, State: Running, Role: LEADER
I20260924 05:09:00.105114 2708 consensus_queue.cc:237] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.105782 2115 catalog_manager.cc:5495] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } health_report { overall_health: UNKNOWN } } }
I20260924 05:09:00.120971 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:00.118843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.0.98.2:46389)
0924 05:09:00.118983 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:00.118985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:00.118993 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:00.119036 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:00.119036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:00.119051 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:00.119122 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:00.119237 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.119674 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.120320 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.120493 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.120541 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:00.120673 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:00.120763 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:00.120811 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:00.120850 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 05:09:00.120864 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":92,"threads_started":1}
W20260924 05:09:00.124624 2319 tablet.cc:2369] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:09:00.125159 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:00.123176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44419 (local address 127.0.98.1:33043)
0924 05:09:00.123220 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:00.123222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:00.123234 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:09:00.123279 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:00.123280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:00.123289 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:00.123333 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:00.123437 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.123852 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.124538 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.124704 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.124777 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:00.124965 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:00.124992 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:00.125042 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:00.125101 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 05:09:00.125118 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:09:00.125209 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:00.123048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.98.2:44419)
0924 05:09:00.123184 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:00.123194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:00.123242 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:00.123346 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:00.123347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:00.123348 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:09:00.123390 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:00.123391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.123933 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.123934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:00.124474 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:00.124475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.124827 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.124828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:00.124881 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:00.124892 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:09:00.125051 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:09:00.125052 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:09:00.125086 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 05:09:00.125131 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 05:09:00.127076 2268 raft_consensus.cc:1261] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Refusing update from remote peer fa43c6af00ed46b49b1794ca3484a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:00.127115 2552 raft_consensus.cc:1261] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Refusing update from remote peer fa43c6af00ed46b49b1794ca3484a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:00.127319 2716 consensus_queue.cc:1035] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:00.127424 2708 consensus_queue.cc:1035] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:00.133700 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:00.131700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60437 (local address 127.0.98.2:46389)
0924 05:09:00.131738 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:00.131740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:00.131834 (+ 94us) server_negotiation.cc:407] Connection header received
0924 05:09:00.131877 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:00.131878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:00.131885 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:00.131923 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:00.131986 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.132378 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.133019 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.133263 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.133325 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:00.133472 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:00.133504 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:00.133554 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:00.133613 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 05:09:00.133631 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:09:00.133730 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:00.131658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.98.1:60437)
0924 05:09:00.131775 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:00.131784 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:00.131839 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:00.131925 (+ 86us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:00.131926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:00.131927 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:09:00.131955 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:00.131955 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.132439 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.132440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:00.132951 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:00.132952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:00.133352 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:00.133354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:00.133408 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:00.133411 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:09:00.133566 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:09:00.133567 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:09:00.133611 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:09:00.133654 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":76,"threads_started":1}
I20260924 05:09:00.139985 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:00.140065 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:00.140697 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf: Sending DeleteTablet for 3 replicas of tablet aecabe536c2d4c568290b39df6e5a3d7
I20260924 05:09:00.140910 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet aecabe536c2d4c568290b39df6e5a3d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260924 05:09:00.140993 2693 tablet_replica.cc:330] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: stopping tablet replica
I20260924 05:09:00.140975 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet aecabe536c2d4c568290b39df6e5a3d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35392
I20260924 05:09:00.141036 2693 raft_consensus.cc:2229] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:00.141021 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet aecabe536c2d4c568290b39df6e5a3d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52402
I20260924 05:09:00.141069 2693 raft_consensus.cc:2258] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:00.141108 2695 tablet_replica.cc:330] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: stopping tablet replica
I20260924 05:09:00.141136 2694 tablet_replica.cc:330] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: stopping tablet replica
I20260924 05:09:00.141157 2695 raft_consensus.cc:2229] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:00.141194 2694 raft_consensus.cc:2229] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:00.141227 2695 raft_consensus.cc:2258] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:00.141244 2694 raft_consensus.cc:2258] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:00.141515 2693 ts_tablet_manager.cc:1905] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:00.141523 2695 ts_tablet_manager.cc:1905] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:00.141611 2694 ts_tablet_manager.cc:1905] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:00.143064 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:00.143205 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:00.143224 2693 ts_tablet_manager.cc:1918] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:09:00.143270 2693 log.cc:1196] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/aecabe536c2d4c568290b39df6e5a3d7
I20260924 05:09:00.143384 2695 ts_tablet_manager.cc:1918] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:09:00.143433 2695 log.cc:1196] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/aecabe536c2d4c568290b39df6e5a3d7
I20260924 05:09:00.143501 2693 ts_tablet_manager.cc:1939] T aecabe536c2d4c568290b39df6e5a3d7 P e1fcae7ed09d4fa6986255a87a28d5f2: Deleting consensus metadata
I20260924 05:09:00.143666 2695 ts_tablet_manager.cc:1939] T aecabe536c2d4c568290b39df6e5a3d7 P fa43c6af00ed46b49b1794ca3484a2fb: Deleting consensus metadata
I20260924 05:09:00.144081 2100 catalog_manager.cc:4841] TS e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043): tablet aecabe536c2d4c568290b39df6e5a3d7 (table test-table [id=a80db279d7244cd79240c6c1037ebaa9]) successfully deleted
I20260924 05:09:00.144361 2099 catalog_manager.cc:4841] TS fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389): tablet aecabe536c2d4c568290b39df6e5a3d7 (table test-table [id=a80db279d7244cd79240c6c1037ebaa9]) successfully deleted
I20260924 05:09:00.144356 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc1e012460c4fe595ea8772d738a2b0 (DEFAULT_TABLE table=test-table [id=751f8b2604ce45d4aa1bace523f2038e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:00.144404 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc1e012460c4fe595ea8772d738a2b0 (DEFAULT_TABLE table=test-table [id=751f8b2604ce45d4aa1bace523f2038e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:00.144457 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc1e012460c4fe595ea8772d738a2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:00.144512 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc1e012460c4fe595ea8772d738a2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:00.144549 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc1e012460c4fe595ea8772d738a2b0 (DEFAULT_TABLE table=test-table [id=751f8b2604ce45d4aa1bace523f2038e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:00.144631 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cc1e012460c4fe595ea8772d738a2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:00.145648 2705 tablet_bootstrap.cc:492] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: Bootstrap starting.
I20260924 05:09:00.145677 2709 tablet_bootstrap.cc:492] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2: Bootstrap starting.
I20260924 05:09:00.145706 2704 tablet_bootstrap.cc:492] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: Bootstrap starting.
I20260924 05:09:00.145990 2704 tablet_bootstrap.cc:654] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:00.145990 2709 tablet_bootstrap.cc:654] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:00.146005 2705 tablet_bootstrap.cc:654] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:00.146588 2705 tablet_bootstrap.cc:492] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: No bootstrap required, opened a new log
I20260924 05:09:00.146639 2705 ts_tablet_manager.cc:1397] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:00.146642 2694 ts_tablet_manager.cc:1918] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:09:00.146692 2694 log.cc:1196] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/aecabe536c2d4c568290b39df6e5a3d7
I20260924 05:09:00.146829 2705 raft_consensus.cc:348] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.146903 2705 raft_consensus.cc:374] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:00.146920 2694 ts_tablet_manager.cc:1939] T aecabe536c2d4c568290b39df6e5a3d7 P a4353208709943d985c0f69eedf745fe: Deleting consensus metadata
I20260924 05:09:00.146929 2705 raft_consensus.cc:729] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa43c6af00ed46b49b1794ca3484a2fb, State: Initialized, Role: FOLLOWER
I20260924 05:09:00.146967 2705 consensus_queue.cc:260] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.147080 2705 ts_tablet_manager.cc:1428] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:00.147387 2704 tablet_bootstrap.cc:492] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: No bootstrap required, opened a new log
I20260924 05:09:00.147436 2704 ts_tablet_manager.cc:1397] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:09:00.147466 2099 catalog_manager.cc:4841] TS a4353208709943d985c0f69eedf745fe (127.0.98.3:46861): tablet aecabe536c2d4c568290b39df6e5a3d7 (table test-table [id=a80db279d7244cd79240c6c1037ebaa9]) successfully deleted
I20260924 05:09:00.147553 2704 raft_consensus.cc:348] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.147611 2704 raft_consensus.cc:374] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:00.147637 2704 raft_consensus.cc:729] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4353208709943d985c0f69eedf745fe, State: Initialized, Role: FOLLOWER
I20260924 05:09:00.147694 2709 tablet_bootstrap.cc:492] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2: No bootstrap required, opened a new log
I20260924 05:09:00.147704 2704 consensus_queue.cc:260] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.147737 2709 ts_tablet_manager.cc:1397] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:00.147789 2704 ts_tablet_manager.cc:1428] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:00.147825 2709 raft_consensus.cc:348] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.147881 2709 raft_consensus.cc:374] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:00.147900 2709 raft_consensus.cc:729] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fcae7ed09d4fa6986255a87a28d5f2, State: Initialized, Role: FOLLOWER
I20260924 05:09:00.147943 2709 consensus_queue.cc:260] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.148025 2709 ts_tablet_manager.cc:1428] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260924 05:09:00.165572 2599 tablet.cc:2369] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:09:00.219973 2708 raft_consensus.cc:482] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:00.220088 2708 raft_consensus.cc:504] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.220237 2708 leader_election.cc:288] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043), a4353208709943d985c0f69eedf745fe (127.0.98.3:46861)
I20260924 05:09:00.220441 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc1e012460c4fe595ea8772d738a2b0" candidate_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4353208709943d985c0f69eedf745fe" is_pre_election: true
I20260924 05:09:00.220563 2552 raft_consensus.cc:2454] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa43c6af00ed46b49b1794ca3484a2fb in term 0.
I20260924 05:09:00.220537 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc1e012460c4fe595ea8772d738a2b0" candidate_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" is_pre_election: true
I20260924 05:09:00.220618 2268 raft_consensus.cc:2454] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa43c6af00ed46b49b1794ca3484a2fb in term 0.
I20260924 05:09:00.220749 2343 leader_election.cc:302] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4353208709943d985c0f69eedf745fe, fa43c6af00ed46b49b1794ca3484a2fb; no voters:
I20260924 05:09:00.220844 2708 raft_consensus.cc:2781] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:00.220902 2708 raft_consensus.cc:482] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:00.220930 2708 raft_consensus.cc:3037] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:00.221563 2708 raft_consensus.cc:504] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.221698 2708 leader_election.cc:288] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [CANDIDATE]: Term 1 election: Requested vote from peers e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043), a4353208709943d985c0f69eedf745fe (127.0.98.3:46861)
I20260924 05:09:00.221871 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc1e012460c4fe595ea8772d738a2b0" candidate_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4353208709943d985c0f69eedf745fe"
I20260924 05:09:00.221882 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc1e012460c4fe595ea8772d738a2b0" candidate_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2"
I20260924 05:09:00.221956 2552 raft_consensus.cc:3037] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:00.221966 2268 raft_consensus.cc:3037] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:00.222577 2552 raft_consensus.cc:2454] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa43c6af00ed46b49b1794ca3484a2fb in term 1.
I20260924 05:09:00.222748 2343 leader_election.cc:302] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4353208709943d985c0f69eedf745fe, fa43c6af00ed46b49b1794ca3484a2fb; no voters:
I20260924 05:09:00.222847 2268 raft_consensus.cc:2454] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa43c6af00ed46b49b1794ca3484a2fb in term 1.
I20260924 05:09:00.222867 2708 raft_consensus.cc:2781] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:00.222909 2708 raft_consensus.cc:686] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 LEADER]: Becoming Leader. State: Replica: fa43c6af00ed46b49b1794ca3484a2fb, State: Running, Role: LEADER
I20260924 05:09:00.222965 2708 consensus_queue.cc:237] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } }
I20260924 05:09:00.223626 2113 catalog_manager.cc:5495] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb reported cstate change: term changed from 0 to 1, leader changed from <none> to fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa43c6af00ed46b49b1794ca3484a2fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 } health_report { overall_health: UNKNOWN } } }
I20260924 05:09:00.244343 2268 raft_consensus.cc:1261] T 5cc1e012460c4fe595ea8772d738a2b0 P e1fcae7ed09d4fa6986255a87a28d5f2 [term 1 FOLLOWER]: Refusing update from remote peer fa43c6af00ed46b49b1794ca3484a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:00.244338 2552 raft_consensus.cc:1261] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Refusing update from remote peer fa43c6af00ed46b49b1794ca3484a2fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:00.244573 2708 consensus_queue.cc:1035] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1fcae7ed09d4fa6986255a87a28d5f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:00.244668 2716 consensus_queue.cc:1035] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4353208709943d985c0f69eedf745fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 05:09:00.248745 2678 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 05:09:01.766453 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:01.766561 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:01.767335 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d869ebe89774cccb0878596fbdff2cf: Sending DeleteTablet for 3 replicas of tablet 5cc1e012460c4fe595ea8772d738a2b0
I20260924 05:09:01.767554 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc1e012460c4fe595ea8772d738a2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35392
I20260924 05:09:01.767585 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc1e012460c4fe595ea8772d738a2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52402
I20260924 05:09:01.767748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 2187
I20260924 05:09:01.767748 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc1e012460c4fe595ea8772d738a2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260924 05:09:01.767781 2758 tablet_replica.cc:330] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: stopping tablet replica
I20260924 05:09:01.768791 2757 tablet_replica.cc:330] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: stopping tablet replica
I20260924 05:09:01.768849 2757 raft_consensus.cc:2229] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:01.768906 2757 raft_consensus.cc:2258] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:01.769156 2757 ts_tablet_manager.cc:1905] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:01.767833 2758 raft_consensus.cc:2229] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:01.770063 2758 raft_consensus.cc:2258] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:01.770329 2758 ts_tablet_manager.cc:1905] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:01.771082 2757 ts_tablet_manager.cc:1918] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:09:01.771142 2757 log.cc:1196] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/5cc1e012460c4fe595ea8772d738a2b0
I20260924 05:09:01.771446 2757 ts_tablet_manager.cc:1939] T 5cc1e012460c4fe595ea8772d738a2b0 P fa43c6af00ed46b49b1794ca3484a2fb: Deleting consensus metadata
I20260924 05:09:01.771809 2099 catalog_manager.cc:4841] TS fa43c6af00ed46b49b1794ca3484a2fb (127.0.98.2:46389): tablet 5cc1e012460c4fe595ea8772d738a2b0 (table test-table [id=751f8b2604ce45d4aa1bace523f2038e]) successfully deleted
I20260924 05:09:01.772074 2758 ts_tablet_manager.cc:1918] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 05:09:01.772145 2758 log.cc:1196] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/5cc1e012460c4fe595ea8772d738a2b0
I20260924 05:09:01.772339 2758 ts_tablet_manager.cc:1939] T 5cc1e012460c4fe595ea8772d738a2b0 P a4353208709943d985c0f69eedf745fe: Deleting consensus metadata
I20260924 05:09:01.772749 2099 catalog_manager.cc:4841] TS a4353208709943d985c0f69eedf745fe (127.0.98.3:46861): tablet 5cc1e012460c4fe595ea8772d738a2b0 (table test-table [id=751f8b2604ce45d4aa1bace523f2038e]) successfully deleted
W20260924 05:09:01.774188 2100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33043 (error 108)
I20260924 05:09:01.774461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 2327
I20260924 05:09:01.775122 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:01.774577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.0.1:46374)
0924 05:09:01.775055 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:01.775092 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":53,"threads_started":1}
W20260924 05:09:01.775226 2100 catalog_manager.cc:4584] TS e1fcae7ed09d4fa6986255a87a28d5f2 (127.0.98.1:33043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cc1e012460c4fe595ea8772d738a2b0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33043: connect: Connection refused (error 111)
I20260924 05:09:01.780634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 2467
I20260924 05:09:01.786060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 2083
2026-09-24T05:09:01Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5340 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:01 dist-test-slave-mr2f krb5kdc[2763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:01 dist-test-slave-mr2f krb5kdc[2763](info): set up 2 sockets
Sep 24 05:09:01 dist-test-slave-mr2f krb5kdc[2763](info): commencing operation
krb5kdc: starting...
W20260924 05:09:02.818508 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:02 dist-test-slave-mr2f krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:09:02.927599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37725
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:03.004916 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:03.005115 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:03.005138 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:03.006232 2776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:03.006268 2776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:03.006281 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:03.006294 2776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:03.006305 2776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:03.007731 2776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:03.007907 2776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:03.008068 2776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:03.012612 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:03.012594 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:03.012655 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:03.012807 2776 server_base.cc:1034] running on GCE node
I20260924 05:09:03.013019 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:03.013283 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:03.014420 2776 hybrid_clock.cc:648] HybridClock initialized: now 1790226543014406 us; error 29 us; skew 500 ppm
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:03.016724 2776 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:09:03.017171 2776 webserver.cc:460] Webserver started at http://127.0.98.62:34143/ using document root <none> and password file <none>
I20260924 05:09:03.017400 2776 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:03.017444 2776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:03.017598 2776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:03.018589 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "fc2c110810b348d3a5722cf0f54139d7"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "f334ebbd7b313df542abeb42328c7f3f"
server_key_iv: "ef1f4fce1eb874ba4f2caabd5a9ac9a0"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.018978 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc2c110810b348d3a5722cf0f54139d7"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "f334ebbd7b313df542abeb42328c7f3f"
server_key_iv: "ef1f4fce1eb874ba4f2caabd5a9ac9a0"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.020318 2776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:03.021166 2791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.021381 2776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:03.021482 2776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "fc2c110810b348d3a5722cf0f54139d7"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "f334ebbd7b313df542abeb42328c7f3f"
server_key_iv: "ef1f4fce1eb874ba4f2caabd5a9ac9a0"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.021593 2776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:03.035714 2776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:03.038798 2776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:03.038944 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:03.045215 2776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37725
I20260924 05:09:03.045296 2843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37725 every 8 connection(s)
I20260924 05:09:03.045934 2776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:03.046706 2844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:03.050042 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7: Bootstrap starting.
I20260924 05:09:03.050652 2844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:03.051069 2844 log.cc:824] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:03.051380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 2776
I20260924 05:09:03.051473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:03.051636 392 external_mini_cluster.cc:1428] Setting key d91ec197511b17df6881c16818a65515
I20260924 05:09:03.051832 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7: No bootstrap required, opened a new log
I20260924 05:09:03.054702 2844 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2c110810b348d3a5722cf0f54139d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37725 } }
I20260924 05:09:03.054911 2844 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:03.054972 2844 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2c110810b348d3a5722cf0f54139d7, State: Initialized, Role: FOLLOWER
I20260924 05:09:03.055107 2844 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2c110810b348d3a5722cf0f54139d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37725 } }
I20260924 05:09:03.055186 2844 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:03.055212 2844 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:03.055253 2844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:03.055869 2844 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2c110810b348d3a5722cf0f54139d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37725 } }
I20260924 05:09:03.055972 2844 leader_election.cc:302] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc2c110810b348d3a5722cf0f54139d7; no voters:
I20260924 05:09:03.056128 2844 leader_election.cc:288] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:03.056159 2849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:03.056270 2849 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [term 1 LEADER]: Becoming Leader. State: Replica: fc2c110810b348d3a5722cf0f54139d7, State: Running, Role: LEADER
I20260924 05:09:03.056349 2849 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2c110810b348d3a5722cf0f54139d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37725 } }
I20260924 05:09:03.056396 2844 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:03.056644 2849 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc2c110810b348d3a5722cf0f54139d7. Latest consensus state: current_term: 1 leader_uuid: "fc2c110810b348d3a5722cf0f54139d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2c110810b348d3a5722cf0f54139d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37725 } } }
I20260924 05:09:03.056725 2849 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:03.056653 2850 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc2c110810b348d3a5722cf0f54139d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2c110810b348d3a5722cf0f54139d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37725 } } }
I20260924 05:09:03.056783 2850 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7 [sys.catalog]: This master's current role is: LEADER
W20260924 05:09:03.058131 2864 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:09:03.058434 2864 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:09:03.058501 2854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:03.058683 2854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:03.060840 2854 catalog_manager.cc:1347] Generated new cluster ID: e9660b0448364613ae9468828317c264
I20260924 05:09:03.060904 2854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:03.064713 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:03.052151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.0.98.62:37725)
0924 05:09:03.052357 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:03.052367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:03.052400 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:09:03.053400 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:03.053408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:03.053469 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:03.053617 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:03.054453 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.055862 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.057431 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.058099 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.058205 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:03.060984 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:03.060988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:03.060988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:03.061000 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:03.062970 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:03.063314 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:03.063316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:03.063317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:03.063404 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:03.063584 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:03.063584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:03.063584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:03.063827 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:03.063966 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:03.064404 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 05:09:03.064528 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260924 05:09:03.081296 2854 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:03.081777 2854 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:03.087225 2854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc2c110810b348d3a5722cf0f54139d7: Generated new TSK 0
I20260924 05:09:03.087381 2854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:09:03.117946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37725
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:03.196686 2870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:03.196871 2870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:03.196892 2870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:03.198019 2870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:03.198062 2870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:03.198115 2870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:03.199687 2870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:03.199927 2870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:03.200150 2870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:03.205128 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:03.205128 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:03.205161 2876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:03.205420 2870 server_base.cc:1034] running on GCE node
I20260924 05:09:03.205663 2870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:03.205926 2870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:03.207085 2870 hybrid_clock.cc:648] HybridClock initialized: now 1790226543207069 us; error 29 us; skew 500 ppm
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:03.209576 2870 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:09:03.209993 2870 webserver.cc:460] Webserver started at http://127.0.98.1:37061/ using document root <none> and password file <none>
I20260924 05:09:03.210208 2870 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:03.210274 2870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:03.210424 2870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:03.211504 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "5f596032b34848e0a3de1feacda158d0"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "529e4013a7cf84df8431d60dedd94cb7"
server_key_iv: "a1f81122d9539de728819bb61104340a"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.211884 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5f596032b34848e0a3de1feacda158d0"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "529e4013a7cf84df8431d60dedd94cb7"
server_key_iv: "a1f81122d9539de728819bb61104340a"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.213129 2870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:03.213819 2885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.213976 2870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:03.214071 2870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "5f596032b34848e0a3de1feacda158d0"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "529e4013a7cf84df8431d60dedd94cb7"
server_key_iv: "a1f81122d9539de728819bb61104340a"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.214147 2870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:03.234440 2870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:03.236757 2870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:03.236914 2870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:03.237136 2870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:03.237449 2870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:03.237499 2870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.237538 2870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:03.237565 2870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.243121 2870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38127
I20260924 05:09:03.243162 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38127 every 8 connection(s)
I20260924 05:09:03.243479 2870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:03.252357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 2870
I20260924 05:09:03.252426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:03.252635 392 external_mini_cluster.cc:1428] Setting key 78b46a398de5aef5ae1bfc27c7f3669d
I20260924 05:09:03.252753 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:03.244150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42853 (local address 127.0.98.62:37725)
0924 05:09:03.244215 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:03.244217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:03.244413 (+ 196us) server_negotiation.cc:407] Connection header received
0924 05:09:03.245375 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:03.245376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:03.245388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:03.245441 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:03.246037 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.246542 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.247603 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.247836 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.247921 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:03.249892 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:03.249893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:03.249893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:03.249901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:03.251215 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:03.251658 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:03.251659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:03.251659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:03.251727 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:03.252014 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:03.252015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:03.252015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:03.252131 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:03.252204 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:03.252551 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 05:09:03.252662 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:09:03.252844 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:03.244139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37725 (local address 127.0.98.1:42853)
0924 05:09:03.244294 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:03.244327 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:03.245250 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:03.245522 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:03.245526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:03.245761 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:03.245935 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:03.245939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.246635 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.246636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:03.247508 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:03.247510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.247931 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.247932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:03.248078 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:03.248498 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:03.248506 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:03.249795 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:03.251320 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:03.251326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:03.251328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:03.251573 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:03.251808 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:03.251809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:03.251809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:03.251932 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:03.252207 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:03.252213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:03.252345 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 05:09:03.252467 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 05:09:03.252614 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260924 05:09:03.253412 2999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37725
I20260924 05:09:03.253508 2999 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:03.253711 2999 heartbeater.cc:511] Master 127.0.98.62:37725 requested a full tablet report, sending...
I20260924 05:09:03.254199 2808 ts_manager.cc:194] Registered new tserver with Master: 5f596032b34848e0a3de1feacda158d0 (127.0.98.1:38127)
I20260924 05:09:03.254879 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42853
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:09:03.278262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37725
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:03.355984 3004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:03.356206 3004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:03.356245 3004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:03.357522 3004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:03.357584 3004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:03.357630 3004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:03.359210 3004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:03.359474 3004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:03.359694 3004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:03.364364 3012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:03.364390 3010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:03.364364 3009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:03.364463 3004 server_base.cc:1034] running on GCE node
I20260924 05:09:03.364811 3004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:03.365042 3004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:03.366195 3004 hybrid_clock.cc:648] HybridClock initialized: now 1790226543366191 us; error 36 us; skew 500 ppm
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:03.368508 3004 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:09:03.368935 3004 webserver.cc:460] Webserver started at http://127.0.98.2:41999/ using document root <none> and password file <none>
I20260924 05:09:03.369154 3004 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:03.369217 3004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:03.369365 3004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:03.370365 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "e0262a417e394c3bbddc355b5344f1c2"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "734d3714797d9bcd09c5a943ddff0093"
server_key_iv: "baf18f24fa84b1c715fc840cfb60b460"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.370728 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e0262a417e394c3bbddc355b5344f1c2"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "734d3714797d9bcd09c5a943ddff0093"
server_key_iv: "baf18f24fa84b1c715fc840cfb60b460"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.372028 3004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:03.372716 3019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.372880 3004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:03.372965 3004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "e0262a417e394c3bbddc355b5344f1c2"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "734d3714797d9bcd09c5a943ddff0093"
server_key_iv: "baf18f24fa84b1c715fc840cfb60b460"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.373047 3004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:03.388967 3004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:03.390587 3004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:03.390749 3004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:03.390996 3004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:03.391278 3004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:03.391330 3004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.391363 3004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:03.391389 3004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.397011 3004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40673
I20260924 05:09:03.397058 3132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40673 every 8 connection(s)
I20260924 05:09:03.397333 3004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:03.402709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 3004
I20260924 05:09:03.402868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:03.403060 392 external_mini_cluster.cc:1428] Setting key 59671d3e5357b1e723ef8369f7d52ab9
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:03.406584 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:03.397991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39447 (local address 127.0.98.62:37725)
0924 05:09:03.398061 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:03.398063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:03.398256 (+ 193us) server_negotiation.cc:407] Connection header received
0924 05:09:03.399253 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:03.399254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:03.399266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:03.399327 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:03.399913 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.400424 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.401402 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.401612 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.401695 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:03.403885 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:03.403886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:03.403886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:03.403893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:03.405162 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:03.405570 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:03.405571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:03.405572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:03.405636 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:03.405926 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:03.405928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:03.405928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:03.406075 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:03.406144 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:03.406390 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 05:09:03.406494 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 05:09:03.406733 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:03.397978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37725 (local address 127.0.98.2:39447)
0924 05:09:03.398143 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:03.398175 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:03.399143 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:03.399399 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:03.399401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:03.399629 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:03.399798 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:03.399802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.400515 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.400516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:03.401311 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:03.401313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.401705 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.401706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:03.401823 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:03.402222 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:03.402229 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:03.403792 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:03.405256 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:03.405263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:03.405265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:03.405488 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:03.405703 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:03.405704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:03.405704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:03.405842 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:03.406147 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:03.406152 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:03.406257 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 05:09:03.406366 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 05:09:03.406513 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260924 05:09:03.407151 3133 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37725
I20260924 05:09:03.407236 3133 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:03.407404 3133 heartbeater.cc:511] Master 127.0.98.62:37725 requested a full tablet report, sending...
I20260924 05:09:03.407778 2808 ts_manager.cc:194] Registered new tserver with Master: e0262a417e394c3bbddc355b5344f1c2 (127.0.98.2:40673)
I20260924 05:09:03.408142 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39447
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:09:03.427322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37725
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:03.506958 3138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:03.507189 3138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:03.507225 3138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:03.508358 3138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:03.508422 3138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:03.508466 3138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:03.509910 3138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:03.510167 3138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:03.510371 3138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:03.515056 3144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:03.515040 3143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:03.515122 3138 server_base.cc:1034] running on GCE node
W20260924 05:09:03.515038 3146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:03.515549 3138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:03.515786 3138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:03.516952 3138 hybrid_clock.cc:648] HybridClock initialized: now 1790226543516930 us; error 44 us; skew 500 ppm
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:03.519345 3138 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:09:03.519767 3138 webserver.cc:460] Webserver started at http://127.0.98.3:36815/ using document root <none> and password file <none>
I20260924 05:09:03.519986 3138 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:03.520052 3138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:03.520196 3138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:03.521282 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "03682fbddd7b4c848989cb22b6aa6d07"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "fd55d9ea874324ebe7d262847b746a10"
server_key_iv: "fd01304b14dcdc2cc965013a995bcbb1"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.521631 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "03682fbddd7b4c848989cb22b6aa6d07"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "fd55d9ea874324ebe7d262847b746a10"
server_key_iv: "fd01304b14dcdc2cc965013a995bcbb1"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.522913 3138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:09:03.523586 3153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.523764 3138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:03.523855 3138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "03682fbddd7b4c848989cb22b6aa6d07"
format_stamp: "Formatted at 2026-09-24 05:09:03 on dist-test-slave-mr2f"
server_key: "fd55d9ea874324ebe7d262847b746a10"
server_key_iv: "fd01304b14dcdc2cc965013a995bcbb1"
server_key_version: "encryptionkey@0"
I20260924 05:09:03.523932 3138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:03.533751 3138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:03.535521 3138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:03.535671 3138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:03.535879 3138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:03.536155 3138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:03.536206 3138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.536244 3138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:03.536273 3138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:03.541543 3138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37893
I20260924 05:09:03.541615 3266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37893 every 8 connection(s)
I20260924 05:09:03.541909 3138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:03.550105 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:03.542550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36875 (local address 127.0.98.62:37725)
0924 05:09:03.542589 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:03.542591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:03.542833 (+ 242us) server_negotiation.cc:407] Connection header received
0924 05:09:03.543675 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:03.543676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:03.543685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:03.543740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:03.544302 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.544793 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.545692 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.545877 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.545925 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:03.547928 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:03.547929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:03.547930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:03.547935 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:03.549067 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:03.549403 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:03.549403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:03.549403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:03.549461 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:03.549657 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:03.549657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:03.549658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:03.549746 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:03.549792 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:03.549995 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 05:09:03.550048 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:09:03.550266 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:03.542519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37725 (local address 127.0.98.3:36875)
0924 05:09:03.542662 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:03.542699 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:03.543587 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:03.543801 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:03.543805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:03.544023 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:03.544228 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:03.544232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.544867 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.544868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:03.545623 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:03.545625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:03.545944 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:03.545944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:03.546034 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:03.546520 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:03.546530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:03.547853 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:03.549142 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:03.549147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:03.549148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:03.549342 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:03.549515 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:03.549516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:03.549516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:03.549600 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:03.549797 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:03.549800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:03.549898 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 05:09:03.549972 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 05:09:03.550074 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":94,"threads_started":1}
I20260924 05:09:03.550736 3267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37725
I20260924 05:09:03.550880 3267 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:03.551021 3267 heartbeater.cc:511] Master 127.0.98.62:37725 requested a full tablet report, sending...
I20260924 05:09:03.551265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 3138
I20260924 05:09:03.551362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:03.551414 2808 ts_manager.cc:194] Registered new tserver with Master: 03682fbddd7b4c848989cb22b6aa6d07 (127.0.98.3:37893)
I20260924 05:09:03.551609 392 external_mini_cluster.cc:1428] Setting key d77ff3c0ad690ec1cdf848ae515e403a
I20260924 05:09:03.551914 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36875
I20260924 05:09:03.552563 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 05:09:03.563241 2848 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:03.562517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.0.98.62:37725)
0924 05:09:03.562554 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:03.562556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:03.562671 (+ 115us) server_negotiation.cc:407] Connection header received
0924 05:09:03.562698 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:03.562699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:03.562707 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:03.562751 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:03.563202 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44390: BlockingRecv error: recv got EOF from 127.0.0.1:44390 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:09:03.564118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 2870
I20260924 05:09:03.568725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 3004
I20260924 05:09:03.572942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 3138
I20260924 05:09:03.577154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 2776
2026-09-24T05:09:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1789 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[3279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[3279](info): set up 2 sockets
Sep 24 05:09:03 dist-test-slave-mr2f krb5kdc[3279](info): commencing operation
krb5kdc: starting...
W20260924 05:09:04.606621 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:04 dist-test-slave-mr2f krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:09:04.717882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45453
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:04.795190 3292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:04.795434 3292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:04.795475 3292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:04.796617 3292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:04.796682 3292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:04.796706 3292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:04.796733 3292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:04.796759 3292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:04.798203 3292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:04.798470 3292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:04.798692 3292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:04.803349 3298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:04.803367 3300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:04.803363 3297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:04.803512 3292 server_base.cc:1034] running on GCE node
I20260924 05:09:04.803746 3292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:04.804041 3292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:04.805188 3292 hybrid_clock.cc:648] HybridClock initialized: now 1790226544805183 us; error 30 us; skew 500 ppm
Sep 24 05:09:04 dist-test-slave-mr2f krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:04.807706 3292 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:09:04.808151 3292 webserver.cc:460] Webserver started at http://127.0.98.62:42761/ using document root <none> and password file <none>
I20260924 05:09:04.808367 3292 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:04.808429 3292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:04.808557 3292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:04.809521 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "40c9bd50f28b412dbc560df75f7bfad1"
format_stamp: "Formatted at 2026-09-24 05:09:04 on dist-test-slave-mr2f"
server_key: "5ed6eccf24c3fd6309a1633761668dfe"
server_key_iv: "d9f09f8439c441c3db323c11fcd2fa5e"
server_key_version: "encryptionkey@0"
I20260924 05:09:04.809852 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "40c9bd50f28b412dbc560df75f7bfad1"
format_stamp: "Formatted at 2026-09-24 05:09:04 on dist-test-slave-mr2f"
server_key: "5ed6eccf24c3fd6309a1633761668dfe"
server_key_iv: "d9f09f8439c441c3db323c11fcd2fa5e"
server_key_version: "encryptionkey@0"
I20260924 05:09:04.811148 3292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:04.811820 3307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:04.812009 3292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:04.812106 3292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "40c9bd50f28b412dbc560df75f7bfad1"
format_stamp: "Formatted at 2026-09-24 05:09:04 on dist-test-slave-mr2f"
server_key: "5ed6eccf24c3fd6309a1633761668dfe"
server_key_iv: "d9f09f8439c441c3db323c11fcd2fa5e"
server_key_version: "encryptionkey@0"
I20260924 05:09:04.812179 3292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:04.820824 3292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:04.822405 3292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:04.822546 3292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:04.826361 3292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45453
I20260924 05:09:04.826406 3359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45453 every 8 connection(s)
I20260924 05:09:04.826725 3292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:04.827509 3360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:04.829581 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1: Bootstrap starting.
I20260924 05:09:04.830170 3360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:04.830453 3360 log.cc:824] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:04.831163 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1: No bootstrap required, opened a new log
I20260924 05:09:04.832336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 3292
I20260924 05:09:04.832436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:04.832598 392 external_mini_cluster.cc:1428] Setting key 74fcc6e50ee9d749238b491d4b4ca7d4
I20260924 05:09:04.833201 3360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c9bd50f28b412dbc560df75f7bfad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } }
I20260924 05:09:04.833352 3360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:04.833374 3360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c9bd50f28b412dbc560df75f7bfad1, State: Initialized, Role: FOLLOWER
I20260924 05:09:04.833503 3360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c9bd50f28b412dbc560df75f7bfad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } }
I20260924 05:09:04.833588 3360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:04.833629 3360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:04.833675 3360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:04.834348 3360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c9bd50f28b412dbc560df75f7bfad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } }
I20260924 05:09:04.834481 3360 leader_election.cc:302] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40c9bd50f28b412dbc560df75f7bfad1; no voters:
I20260924 05:09:04.834649 3360 leader_election.cc:288] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:04.834702 3365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:04.834911 3360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:04.834909 3365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [term 1 LEADER]: Becoming Leader. State: Replica: 40c9bd50f28b412dbc560df75f7bfad1, State: Running, Role: LEADER
I20260924 05:09:04.835033 3365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c9bd50f28b412dbc560df75f7bfad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } }
I20260924 05:09:04.835314 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40c9bd50f28b412dbc560df75f7bfad1. Latest consensus state: current_term: 1 leader_uuid: "40c9bd50f28b412dbc560df75f7bfad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c9bd50f28b412dbc560df75f7bfad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } }
I20260924 05:09:04.835393 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:04.835328 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40c9bd50f28b412dbc560df75f7bfad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c9bd50f28b412dbc560df75f7bfad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45453 } } }
I20260924 05:09:04.835794 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:04.836216 3374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:04.836751 3374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:04.838176 3374 catalog_manager.cc:1347] Generated new cluster ID: 330b1209d592404b8f665ecae3fcc42a
I20260924 05:09:04.838243 3374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:09:04 dist-test-slave-mr2f krb5kdc[3279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:04.844201 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:04.833086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.0.98.62:45453)
0924 05:09:04.833586 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:04.833595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:04.833624 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:09:04.833732 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:04.833738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:04.833794 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:04.833895 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:04.834718 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:04.836150 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:04.837226 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:04.837723 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:04.837876 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:04.840547 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:04.840551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:04.840551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:04.840563 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:04.842547 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:04.842935 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:04.842938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:04.842939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:04.843057 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:04.843274 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:04.843274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:04.843275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:04.843579 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:04.843760 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:04.843827 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 05:09:04.843965 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":436,"threads_started":1}
I20260924 05:09:04.845198 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:09:04.846007 3374 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:04.846657 3374 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:04.851630 3374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40c9bd50f28b412dbc560df75f7bfad1: Generated new TSK 0
I20260924 05:09:04.851804 3374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:09:04.863286 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:04.859622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.0.98.62:45453)
0924 05:09:04.859667 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:04.859669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:04.859854 (+ 185us) server_negotiation.cc:407] Connection header received
0924 05:09:04.859936 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:04.859937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:04.859949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:04.860029 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:04.860402 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:04.860903 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:04.861661 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:04.861884 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:04.861985 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:04.862788 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:04.862789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:04.862789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:04.862989 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:04.863005 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:04.863011 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:04.863082 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:04.863171 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 05:09:04.863195 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:04.865787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 3292
2026-09-24T05:09:04Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1286 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:04 dist-test-slave-mr2f krb5kdc[3393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:04 dist-test-slave-mr2f krb5kdc[3393](info): set up 2 sockets
Sep 24 05:09:04 dist-test-slave-mr2f krb5kdc[3393](info): commencing operation
krb5kdc: starting...
W20260924 05:09:05.882572 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:05 dist-test-slave-mr2f krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:09:05.998428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38217
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:06.077109 3406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:06.077330 3406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:06.077368 3406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:06.078485 3406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:06.078547 3406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:06.078572 3406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:06.078603 3406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:06.078629 3406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:06.080121 3406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:06.080387 3406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:06.080602 3406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:06.085352 3411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:06.085450 3406 server_base.cc:1034] running on GCE node
W20260924 05:09:06.085345 3412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:06.085345 3414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:06.085834 3406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:06.086148 3406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:06.087304 3406 hybrid_clock.cc:648] HybridClock initialized: now 1790226546087286 us; error 31 us; skew 500 ppm
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:06.094379 3406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:09:06.094905 3406 webserver.cc:460] Webserver started at http://127.0.98.62:36615/ using document root <none> and password file <none>
I20260924 05:09:06.095137 3406 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:06.095207 3406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:06.095363 3406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:06.096446 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "518ce1144bac409c8ef7d8026ec3ae88"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "154e2b1eee9c0e36743c19a72da53271"
server_key_iv: "50decf72d9f092e865c5b7edd7c846e0"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.096835 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "518ce1144bac409c8ef7d8026ec3ae88"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "154e2b1eee9c0e36743c19a72da53271"
server_key_iv: "50decf72d9f092e865c5b7edd7c846e0"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.098132 3406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:06.098920 3421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.099086 3406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:06.099188 3406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "518ce1144bac409c8ef7d8026ec3ae88"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "154e2b1eee9c0e36743c19a72da53271"
server_key_iv: "50decf72d9f092e865c5b7edd7c846e0"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.099277 3406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:06.128947 3406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:06.130713 3406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:06.130897 3406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:06.134616 3406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38217
I20260924 05:09:06.134653 3473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38217 every 8 connection(s)
I20260924 05:09:06.135042 3406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:06.135979 3474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:06.138080 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88: Bootstrap starting.
I20260924 05:09:06.138581 3474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:06.138890 3474 log.cc:824] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:06.139518 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88: No bootstrap required, opened a new log
I20260924 05:09:06.141001 3474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ce1144bac409c8ef7d8026ec3ae88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38217 } }
I20260924 05:09:06.141134 3474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:06.141156 3474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518ce1144bac409c8ef7d8026ec3ae88, State: Initialized, Role: FOLLOWER
I20260924 05:09:06.141256 3474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ce1144bac409c8ef7d8026ec3ae88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38217 } }
I20260924 05:09:06.141337 3474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:06.141371 3474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:06.141426 3474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:06.142215 3474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ce1144bac409c8ef7d8026ec3ae88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38217 } }
I20260924 05:09:06.142336 3474 leader_election.cc:302] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 518ce1144bac409c8ef7d8026ec3ae88; no voters:
I20260924 05:09:06.142513 3474 leader_election.cc:288] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:06.142560 3477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:06.142793 3474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:06.142827 3477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [term 1 LEADER]: Becoming Leader. State: Replica: 518ce1144bac409c8ef7d8026ec3ae88, State: Running, Role: LEADER
I20260924 05:09:06.142905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 3406
I20260924 05:09:06.143002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:06.143059 3477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ce1144bac409c8ef7d8026ec3ae88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38217 } }
I20260924 05:09:06.143182 392 external_mini_cluster.cc:1428] Setting key 3f640134c4b6241c5e16338d078f185b
I20260924 05:09:06.143447 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "518ce1144bac409c8ef7d8026ec3ae88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ce1144bac409c8ef7d8026ec3ae88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38217 } } }
I20260924 05:09:06.143543 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:06.143703 3479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 518ce1144bac409c8ef7d8026ec3ae88. Latest consensus state: current_term: 1 leader_uuid: "518ce1144bac409c8ef7d8026ec3ae88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518ce1144bac409c8ef7d8026ec3ae88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38217 } } }
I20260924 05:09:06.143752 3479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:06.144209 3486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:06.144994 3486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:06.146426 3486 catalog_manager.cc:1347] Generated new cluster ID: 1989ee1d63234358aa1ea39f28484f4f
I20260924 05:09:06.146476 3486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:06.153512 3486 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:06.154148 3486 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:06.154976 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:06.143860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47072 (local address 127.0.98.62:38217)
0924 05:09:06.144044 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:06.144054 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:06.144085 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:09:06.144207 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:06.144215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:06.144379 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:06.144547 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:06.145350 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.146669 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.147749 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.148371 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.148483 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:06.151284 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:06.151289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:06.151289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:06.151302 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:06.153329 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.153711 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.153715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.153715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.153826 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.154051 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.154052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.154052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.154325 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:06.154486 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:06.154586 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 05:09:06.154731 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260924 05:09:06.158471 3486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 518ce1144bac409c8ef7d8026ec3ae88: Generated new TSK 0
I20260924 05:09:06.158638 3486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:09:06.189673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38217
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:06.273191 3500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:06.273383 3500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:06.273406 3500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:06.274569 3500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:06.274612 3500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:06.274639 3500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:06.276067 3500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:06.276243 3500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:06.276412 3500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:06.281152 3508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:06.281265 3500 server_base.cc:1034] running on GCE node
W20260924 05:09:06.281205 3505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:06.281221 3506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:06.281596 3500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:06.281847 3500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:06.283012 3500 hybrid_clock.cc:648] HybridClock initialized: now 1790226546283001 us; error 43 us; skew 500 ppm
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:06.285368 3500 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:09:06.285779 3500 webserver.cc:460] Webserver started at http://127.0.98.1:35339/ using document root <none> and password file <none>
I20260924 05:09:06.285996 3500 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:06.286063 3500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:06.286216 3500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:06.287303 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "61a893a3ee2f43348187fe354668eed4"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "fd88fb5647ff8c073a87cb68c736987e"
server_key_iv: "c7da0cffc5c271194eee074dd08148f0"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.287650 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "61a893a3ee2f43348187fe354668eed4"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "fd88fb5647ff8c073a87cb68c736987e"
server_key_iv: "c7da0cffc5c271194eee074dd08148f0"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.288920 3500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:06.289601 3515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.289769 3500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:06.289873 3500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "61a893a3ee2f43348187fe354668eed4"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "fd88fb5647ff8c073a87cb68c736987e"
server_key_iv: "c7da0cffc5c271194eee074dd08148f0"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.289945 3500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:06.306167 3500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:06.308024 3500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:06.308183 3500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:06.308409 3500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:06.308714 3500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:06.308768 3500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.308809 3500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:06.308838 3500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.314374 3500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36083
I20260924 05:09:06.314442 3628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36083 every 8 connection(s)
I20260924 05:09:06.314792 3500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:06.323319 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:06.315539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53181 (local address 127.0.98.62:38217)
0924 05:09:06.315588 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:06.315591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:06.315770 (+ 179us) server_negotiation.cc:407] Connection header received
0924 05:09:06.316269 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:06.316270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:06.316281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:06.316340 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:06.317218 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.317705 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.318629 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.318886 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.318940 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:06.320820 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:06.320822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:06.320822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:06.320828 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:06.322194 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.322519 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.322519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.322520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.322570 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.322817 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.322818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.322818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.322916 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:06.322968 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:06.323188 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 05:09:06.323255 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:09:06.323482 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:06.315489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38217 (local address 127.0.98.1:53181)
0924 05:09:06.315647 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:06.315696 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:06.316166 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:06.316394 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:06.316397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:06.316951 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:06.317132 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:06.317137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.317785 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.317786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:06.318551 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:06.318553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.318955 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.318956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:06.319051 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:06.319466 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:06.319473 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:06.320735 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:06.322261 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:06.322266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:06.322267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:06.322456 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:06.322626 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:06.322626 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:06.322626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:06.322719 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:06.322972 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:06.322976 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:06.323080 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 05:09:06.323159 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 05:09:06.323268 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260924 05:09:06.323990 3629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38217
I20260924 05:09:06.324083 3629 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:06.324285 3629 heartbeater.cc:511] Master 127.0.98.62:38217 requested a full tablet report, sending...
I20260924 05:09:06.324349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 3500
I20260924 05:09:06.324448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:06.324648 392 external_mini_cluster.cc:1428] Setting key d7a2d17c6dd5a62d10ade142ed1cb254
I20260924 05:09:06.324842 3438 ts_manager.cc:194] Registered new tserver with Master: 61a893a3ee2f43348187fe354668eed4 (127.0.98.1:36083)
I20260924 05:09:06.325835 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53181
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:09:06.347591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38217
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:06.428423 3634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:06.428608 3634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:06.428632 3634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:06.429651 3634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:06.429682 3634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:06.429706 3634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:06.430990 3634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:06.431156 3634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:06.431311 3634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:06.435882 3639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:06.435871 3640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:06.435871 3642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:06.436121 3634 server_base.cc:1034] running on GCE node
I20260924 05:09:06.436299 3634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:06.436529 3634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:06.437666 3634 hybrid_clock.cc:648] HybridClock initialized: now 1790226546437656 us; error 32 us; skew 500 ppm
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:06.440106 3634 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:09:06.440500 3634 webserver.cc:460] Webserver started at http://127.0.98.2:40349/ using document root <none> and password file <none>
I20260924 05:09:06.440729 3634 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:06.440773 3634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:06.440917 3634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:06.442010 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "541ad01c755b45348397bfa4efa85b2c"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "d77b132657c985b76d4d03f135e4be5f"
server_key_iv: "1ee62e322ee75ac146f860d0317510ff"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.442348 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "541ad01c755b45348397bfa4efa85b2c"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "d77b132657c985b76d4d03f135e4be5f"
server_key_iv: "1ee62e322ee75ac146f860d0317510ff"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.443872 3634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:06.444608 3649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.444800 3634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:06.444870 3634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "541ad01c755b45348397bfa4efa85b2c"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "d77b132657c985b76d4d03f135e4be5f"
server_key_iv: "1ee62e322ee75ac146f860d0317510ff"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.444948 3634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:06.458567 3634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:06.460333 3634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:06.460486 3634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:06.460702 3634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:06.461005 3634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:06.461059 3634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.461102 3634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:06.461129 3634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.466919 3634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37357
I20260924 05:09:06.467000 3762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37357 every 8 connection(s)
I20260924 05:09:06.467301 3634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:06.472261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 3634
I20260924 05:09:06.472390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:06.472580 392 external_mini_cluster.cc:1428] Setting key fd51390c7de3af9d476729db1fce9475
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:06.476758 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:06.468089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37157 (local address 127.0.98.62:38217)
0924 05:09:06.468167 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:06.468169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:06.468334 (+ 165us) server_negotiation.cc:407] Connection header received
0924 05:09:06.468794 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:06.468795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:06.468803 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:06.468860 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:06.469724 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.470207 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.471218 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.471392 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.471444 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:06.473717 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:06.473719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:06.473719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:06.473727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:06.475296 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.475803 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.475804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.475805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.475866 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.476095 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.476096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.476096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.476253 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:06.476323 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:06.476552 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 05:09:06.476640 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 05:09:06.476814 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:06.468043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38217 (local address 127.0.98.2:37157)
0924 05:09:06.468205 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:06.468244 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:06.468709 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:06.468911 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:06.468914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:06.469462 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:06.469653 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:06.469657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.470283 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.470283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:06.471147 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:06.471149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.471462 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.471462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:06.471561 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:06.472028 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:06.472035 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:06.473625 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:06.475396 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:06.475402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:06.475404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:06.475723 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:06.475934 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:06.475934 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:06.475935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:06.476026 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:06.476317 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:06.476320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:06.476425 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 05:09:06.476503 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 05:09:06.476616 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260924 05:09:06.477377 3763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38217
I20260924 05:09:06.477478 3763 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:06.477675 3763 heartbeater.cc:511] Master 127.0.98.62:38217 requested a full tablet report, sending...
I20260924 05:09:06.478040 3438 ts_manager.cc:194] Registered new tserver with Master: 541ad01c755b45348397bfa4efa85b2c (127.0.98.2:37357)
I20260924 05:09:06.478430 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37157
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:09:06.497195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38217
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:06.574988 3768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:06.575228 3768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:06.575266 3768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:06.576424 3768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:06.576495 3768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:06.576540 3768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:06.577970 3768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:06.578230 3768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:06.578452 3768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:06.583113 3776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:06.583135 3774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:06.583184 3768 server_base.cc:1034] running on GCE node
W20260924 05:09:06.583151 3773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:06.583398 3768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:06.583644 3768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:06.584797 3768 hybrid_clock.cc:648] HybridClock initialized: now 1790226546584784 us; error 29 us; skew 500 ppm
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:06.587087 3768 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:09:06.587502 3768 webserver.cc:460] Webserver started at http://127.0.98.3:36913/ using document root <none> and password file <none>
I20260924 05:09:06.587697 3768 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:06.587736 3768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:06.587819 3768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:06.588730 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9be26ec575f4fbd996faedd741cbfe5"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "86534ddc92c9c10f872bd18240055a3d"
server_key_iv: "1e608b39611315a60bd76afc2406b505"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.589007 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9be26ec575f4fbd996faedd741cbfe5"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "86534ddc92c9c10f872bd18240055a3d"
server_key_iv: "1e608b39611315a60bd76afc2406b505"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.590197 3768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:06.590852 3783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.591032 3768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:06.591131 3768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "f9be26ec575f4fbd996faedd741cbfe5"
format_stamp: "Formatted at 2026-09-24 05:09:06 on dist-test-slave-mr2f"
server_key: "86534ddc92c9c10f872bd18240055a3d"
server_key_iv: "1e608b39611315a60bd76afc2406b505"
server_key_version: "encryptionkey@0"
I20260924 05:09:06.591202 3768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:06.612919 3768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:06.614683 3768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:06.614858 3768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:06.615074 3768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:06.615377 3768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:06.615415 3768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.615437 3768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:06.615499 3768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:06.620932 3768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43235
I20260924 05:09:06.621021 3896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43235 every 8 connection(s)
I20260924 05:09:06.621368 3768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:06.629500 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:06.622037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40785 (local address 127.0.98.62:38217)
0924 05:09:06.622106 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:06.622108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:06.622301 (+ 193us) server_negotiation.cc:407] Connection header received
0924 05:09:06.622727 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:06.622729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:06.622738 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:06.622848 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:06.623701 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.624156 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.625108 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.625283 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.625332 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:06.627176 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:06.627177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:06.627177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:06.627183 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:06.628442 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.628791 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.628792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.628792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.628841 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.629035 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.629035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.629036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.629125 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:06.629172 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:06.629385 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 05:09:06.629439 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 05:09:06.629694 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:06.622021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38217 (local address 127.0.98.3:40785)
0924 05:09:06.622186 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:06.622219 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:06.622649 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:06.622917 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:06.622920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:06.623443 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:06.623620 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:06.623624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.624232 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.624232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:06.625023 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:06.625025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.625349 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.625350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:06.625442 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:06.625853 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:06.625860 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:06.627093 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:06.628510 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:06.628515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:06.628516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:06.628728 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:06.628896 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:06.628896 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:06.628896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:06.628983 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:06.629177 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:06.629181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:06.629281 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 05:09:06.629358 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 05:09:06.629465 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260924 05:09:06.630286 3897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38217
I20260924 05:09:06.630391 3897 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:06.630558 3897 heartbeater.cc:511] Master 127.0.98.62:38217 requested a full tablet report, sending...
I20260924 05:09:06.630963 3438 ts_manager.cc:194] Registered new tserver with Master: f9be26ec575f4fbd996faedd741cbfe5 (127.0.98.3:43235)
I20260924 05:09:06.631341 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40785
I20260924 05:09:06.631541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 3768
I20260924 05:09:06.631644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:06.631861 392 external_mini_cluster.cc:1428] Setting key ac7967f6b8e3eb25ad01fba86a2f7017
I20260924 05:09:06.632589 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226546, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226546, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:06.654177 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:06.647489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.0.98.62:38217)
0924 05:09:06.647537 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:06.647539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:06.647681 (+ 142us) server_negotiation.cc:407] Connection header received
0924 05:09:06.647722 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:06.647723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:06.647731 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:06.647778 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:06.648306 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.648814 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.649497 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:06.649671 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:06.649717 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:06.651914 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:06.651915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:06.651916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:06.651921 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:06.653270 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.653542 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.653543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.653543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.653588 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:06.653736 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:06.653737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:06.653737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:06.653866 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:06.653933 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:06.654022 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 05:09:06.654073 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 05:09:06.654425 3438 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47088
I20260924 05:09:06.654887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 3500
I20260924 05:09:06.659284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 3634
I20260924 05:09:06.663923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 3768
I20260924 05:09:06.668141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 3406
2026-09-24T05:09:06Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1803 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3909](info): set up 2 sockets
Sep 24 05:09:06 dist-test-slave-mr2f krb5kdc[3909](info): commencing operation
krb5kdc: starting...
W20260924 05:09:07.678572 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:07 dist-test-slave-mr2f krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:09:07.789752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:07.867988 3922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:07.868184 3922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:07.868206 3922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:07.869359 3922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:07.869403 3922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:07.869418 3922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:07.869431 3922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:07.869443 3922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:07.870916 3922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:07.871176 3922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:07.871403 3922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:07.876096 3930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:07.876148 3928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:07.876148 3927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:07.876407 3922 server_base.cc:1034] running on GCE node
I20260924 05:09:07.876641 3922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:07.876874 3922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:07.878012 3922 hybrid_clock.cc:648] HybridClock initialized: now 1790226547878001 us; error 33 us; skew 500 ppm
Sep 24 05:09:07 dist-test-slave-mr2f krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:07.880481 3922 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:09:07.880880 3922 webserver.cc:460] Webserver started at http://127.0.98.62:44561/ using document root <none> and password file <none>
I20260924 05:09:07.881072 3922 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:07.881114 3922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:07.881199 3922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:07.882166 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "f8b746d19e6140c8b85e461f8e93563d"
format_stamp: "Formatted at 2026-09-24 05:09:07 on dist-test-slave-mr2f"
server_key: "e30d8064022527f5a827f22c7d502423"
server_key_iv: "e9d0792be5c2245acd26cf5e41c799d1"
server_key_version: "encryptionkey@0"
I20260924 05:09:07.882453 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8b746d19e6140c8b85e461f8e93563d"
format_stamp: "Formatted at 2026-09-24 05:09:07 on dist-test-slave-mr2f"
server_key: "e30d8064022527f5a827f22c7d502423"
server_key_iv: "e9d0792be5c2245acd26cf5e41c799d1"
server_key_version: "encryptionkey@0"
I20260924 05:09:07.883769 3922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:07.884419 3937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:07.884593 3922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:07.884719 3922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "f8b746d19e6140c8b85e461f8e93563d"
format_stamp: "Formatted at 2026-09-24 05:09:07 on dist-test-slave-mr2f"
server_key: "e30d8064022527f5a827f22c7d502423"
server_key_iv: "e9d0792be5c2245acd26cf5e41c799d1"
server_key_version: "encryptionkey@0"
I20260924 05:09:07.884791 3922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:07.904826 3922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:07.906699 3922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:07.906884 3922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:07.910491 3922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34377
I20260924 05:09:07.910583 3989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34377 every 8 connection(s)
I20260924 05:09:07.910949 3922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:07.911761 3990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:07.913851 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d: Bootstrap starting.
I20260924 05:09:07.914050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 3922
I20260924 05:09:07.914172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:07.914337 392 external_mini_cluster.cc:1428] Setting key c927aa4e280f0ddf820dd806577a0e09
I20260924 05:09:07.914531 3990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:07.914901 3990 log.cc:824] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:07.915728 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d: No bootstrap required, opened a new log
I20260924 05:09:07.917972 3990 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b746d19e6140c8b85e461f8e93563d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34377 } }
I20260924 05:09:07.918131 3990 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:07.918161 3990 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b746d19e6140c8b85e461f8e93563d, State: Initialized, Role: FOLLOWER
I20260924 05:09:07.918275 3990 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b746d19e6140c8b85e461f8e93563d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34377 } }
I20260924 05:09:07.918361 3990 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:07.918396 3990 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:07.918444 3990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:07.919087 3990 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b746d19e6140c8b85e461f8e93563d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34377 } }
I20260924 05:09:07.919198 3990 leader_election.cc:302] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8b746d19e6140c8b85e461f8e93563d; no voters:
I20260924 05:09:07.919355 3990 leader_election.cc:288] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:07.919415 3995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:07.919576 3995 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [term 1 LEADER]: Becoming Leader. State: Replica: f8b746d19e6140c8b85e461f8e93563d, State: Running, Role: LEADER
I20260924 05:09:07.919627 3990 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:07.919677 3995 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b746d19e6140c8b85e461f8e93563d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34377 } }
I20260924 05:09:07.919963 3996 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8b746d19e6140c8b85e461f8e93563d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b746d19e6140c8b85e461f8e93563d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34377 } } }
I20260924 05:09:07.920069 3996 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:07.919950 3995 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b746d19e6140c8b85e461f8e93563d. Latest consensus state: current_term: 1 leader_uuid: "f8b746d19e6140c8b85e461f8e93563d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b746d19e6140c8b85e461f8e93563d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34377 } } }
I20260924 05:09:07.920289 3995 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:07.920750 4001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:07.921239 4001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:09:07 dist-test-slave-mr2f krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:07.922679 4001 catalog_manager.cc:1347] Generated new cluster ID: e55ac6de4f574baaa970044142e0d4e2
I20260924 05:09:07.922734 4001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:07.925421 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:07.914867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.0.98.62:34377)
0924 05:09:07.915029 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:07.915044 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:07.915069 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:09:07.915447 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:07.915453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:07.915509 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:07.915635 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:07.916409 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:07.917290 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:07.918378 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:07.918888 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:07.918969 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:07.921925 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:07.921929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:07.921930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:07.921940 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:07.923867 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:07.924225 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:07.924228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:07.924228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:07.924331 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:07.924524 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:07.924525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:07.924525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:07.924838 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:07.924995 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:07.925081 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 05:09:07.925215 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260924 05:09:07.937745 4001 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:07.938444 4001 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:07.942302 4001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b746d19e6140c8b85e461f8e93563d: Generated new TSK 0
I20260924 05:09:07.942471 4001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:09:07.977653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34377
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:08.067318 4016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:08.067507 4016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:08.067531 4016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:08.068668 4016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:08.068712 4016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:08.068746 4016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:08.070197 4016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:08.070442 4016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:08.070683 4016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:08.075595 4024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:08.075652 4022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:08.075748 4016 server_base.cc:1034] running on GCE node
W20260924 05:09:08.075654 4021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:08.076068 4016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:08.076325 4016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:08.077491 4016 hybrid_clock.cc:648] HybridClock initialized: now 1790226548077469 us; error 39 us; skew 500 ppm
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:08.079959 4016 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:09:08.080377 4016 webserver.cc:460] Webserver started at http://127.0.98.1:43449/ using document root <none> and password file <none>
I20260924 05:09:08.080626 4016 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:08.080696 4016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:08.080837 4016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:08.081827 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "751955b066b0405ab7cd20454f32ea29"
format_stamp: "Formatted at 2026-09-24 05:09:08 on dist-test-slave-mr2f"
server_key: "68cb6d165ba1aba55c4c38f20bb072ee"
server_key_iv: "ac08f161d06b23b769314c4f1a09f796"
server_key_version: "encryptionkey@0"
I20260924 05:09:08.082207 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "751955b066b0405ab7cd20454f32ea29"
format_stamp: "Formatted at 2026-09-24 05:09:08 on dist-test-slave-mr2f"
server_key: "68cb6d165ba1aba55c4c38f20bb072ee"
server_key_iv: "ac08f161d06b23b769314c4f1a09f796"
server_key_version: "encryptionkey@0"
I20260924 05:09:08.083556 4016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:08.084343 4031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:08.084528 4016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:08.084640 4016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "751955b066b0405ab7cd20454f32ea29"
format_stamp: "Formatted at 2026-09-24 05:09:08 on dist-test-slave-mr2f"
server_key: "68cb6d165ba1aba55c4c38f20bb072ee"
server_key_iv: "ac08f161d06b23b769314c4f1a09f796"
server_key_version: "encryptionkey@0"
I20260924 05:09:08.084723 4016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:08.104506 4016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:08.106276 4016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:08.106433 4016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:08.106664 4016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:08.107000 4016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:08.107055 4016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:08.107087 4016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:08.107117 4016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:08.112591 4016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44741
I20260924 05:09:08.112618 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44741 every 8 connection(s)
I20260924 05:09:08.112931 4016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:08.121905 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:08.113656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55193 (local address 127.0.98.62:34377)
0924 05:09:08.113718 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:08.113720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:08.113907 (+ 187us) server_negotiation.cc:407] Connection header received
0924 05:09:08.114860 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:08.114861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:08.114872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:08.114935 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:08.115518 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.116008 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.116943 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.117131 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.117182 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:08.119181 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:08.119183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:08.119183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:08.119189 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:08.120705 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.121112 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.121113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.121113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.121162 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.121367 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.121367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.121368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.121491 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:08.121555 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:08.121760 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 05:09:08.121842 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:09:08.122038 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:08.113610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34377 (local address 127.0.98.1:55193)
0924 05:09:08.113780 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:08.113820 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:08.114680 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:08.115008 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:08.115011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:08.115256 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:08.115432 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:08.115436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.116083 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.116084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:08.116870 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:08.116871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.117200 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.117200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:08.117297 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:08.117746 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:08.117753 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:08.119104 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:08.120807 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:08.120811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:08.120813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:08.121049 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:08.121220 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:08.121220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:08.121221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:08.121312 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:08.121550 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:08.121553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:08.121655 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 05:09:08.121740 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 05:09:08.121860 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260924 05:09:08.122519 4145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34377
I20260924 05:09:08.122619 4145 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:08.122795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 4016
I20260924 05:09:08.122828 4145 heartbeater.cc:511] Master 127.0.98.62:34377 requested a full tablet report, sending...
I20260924 05:09:08.122905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:08.123128 392 external_mini_cluster.cc:1428] Setting key 42e1473c718b818f766612d8219a58c4
I20260924 05:09:08.123425 3954 ts_manager.cc:194] Registered new tserver with Master: 751955b066b0405ab7cd20454f32ea29 (127.0.98.1:44741)
I20260924 05:09:08.124148 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55193
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:09:08.145660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34377
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:08.225122 4150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:08.225344 4150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:08.225384 4150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:08.226573 4150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:08.226639 4150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:08.226684 4150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:08.228155 4150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:08.228410 4150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:08.228637 4150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:08.233433 4158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:08.233507 4156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:08.233639 4155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:08.234843 4150 server_base.cc:1034] running on GCE node
I20260924 05:09:08.235031 4150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:08.235244 4150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:08.236373 4150 hybrid_clock.cc:648] HybridClock initialized: now 1790226548236352 us; error 36 us; skew 500 ppm
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:08.239432 4150 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:09:08.239955 4150 webserver.cc:460] Webserver started at http://127.0.98.2:45053/ using document root <none> and password file <none>
I20260924 05:09:08.240195 4150 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:08.240268 4150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:08.240415 4150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:08.241753 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "20bade09db274e75a3cb57cbbca42231"
format_stamp: "Formatted at 2026-09-24 05:09:08 on dist-test-slave-mr2f"
server_key: "cddd07e5f78baf86eb93b916771bef1a"
server_key_iv: "60256a0c31c9895ac173be57467ad524"
server_key_version: "encryptionkey@0"
I20260924 05:09:08.242137 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "20bade09db274e75a3cb57cbbca42231"
format_stamp: "Formatted at 2026-09-24 05:09:08 on dist-test-slave-mr2f"
server_key: "cddd07e5f78baf86eb93b916771bef1a"
server_key_iv: "60256a0c31c9895ac173be57467ad524"
server_key_version: "encryptionkey@0"
I20260924 05:09:08.243907 4150 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260924 05:09:08.244678 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:08.244846 4150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:08.244946 4150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "20bade09db274e75a3cb57cbbca42231"
format_stamp: "Formatted at 2026-09-24 05:09:08 on dist-test-slave-mr2f"
server_key: "cddd07e5f78baf86eb93b916771bef1a"
server_key_iv: "60256a0c31c9895ac173be57467ad524"
server_key_version: "encryptionkey@0"
I20260924 05:09:08.245034 4150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:08.258453 4150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:08.260320 4150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:08.260473 4150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:08.260684 4150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:08.260982 4150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:08.261036 4150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:08.261085 4150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:08.261125 4150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:08.266667 4150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34635
I20260924 05:09:08.266745 4278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34635 every 8 connection(s)
I20260924 05:09:08.267077 4150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:08.269937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 4150
I20260924 05:09:08.270041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:08.270231 392 external_mini_cluster.cc:1428] Setting key e7f72dcfdda185acc1b9933c5d31c530
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:08.277182 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:08.267791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42925 (local address 127.0.98.62:34377)
0924 05:09:08.267848 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:08.267850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:08.268027 (+ 177us) server_negotiation.cc:407] Connection header received
0924 05:09:08.268981 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:08.268982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:08.268991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:08.269044 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:08.269595 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.270342 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.271491 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.271829 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.271902 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:08.274277 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:08.274279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:08.274279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:08.274287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:08.275728 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.276240 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.276241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.276241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.276290 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.276583 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.276584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.276584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.276676 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:08.276725 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:08.277060 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 05:09:08.277120 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:09:08.277464 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:08.267746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34377 (local address 127.0.98.2:42925)
0924 05:09:08.267904 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:08.267944 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:08.268889 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:08.269104 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:08.269108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:08.269346 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:08.269524 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:08.269528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.270444 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.270446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:08.271403 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:08.271406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.271910 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.271911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:08.272009 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:08.272635 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:08.272642 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:08.274175 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:08.275813 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:08.275820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:08.275823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:08.276153 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:08.276354 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:08.276355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:08.276355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:08.276497 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:08.276747 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:08.276754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:08.276925 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 05:09:08.277044 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 05:09:08.277212 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260924 05:09:08.278012 4279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34377
I20260924 05:09:08.278102 4279 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:08.278294 4279 heartbeater.cc:511] Master 127.0.98.62:34377 requested a full tablet report, sending...
I20260924 05:09:08.278677 3954 ts_manager.cc:194] Registered new tserver with Master: 20bade09db274e75a3cb57cbbca42231 (127.0.98.2:34635)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 05:09:08.279088 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:09:08.295595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34377
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:08.373517 4284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:08.373760 4284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:08.373799 4284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:08.374967 4284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:08.375033 4284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:08.375077 4284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:08.376564 4284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:08.376827 4284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:08.377046 4284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:08.381912 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:08.381917 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:08.382001 4284 server_base.cc:1034] running on GCE node
W20260924 05:09:08.381927 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:08.382365 4284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:08.382601 4284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:08.383761 4284 hybrid_clock.cc:648] HybridClock initialized: now 1790226548383746 us; error 29 us; skew 500 ppm
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:08.386188 4284 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:09:08.386642 4284 webserver.cc:460] Webserver started at http://127.0.98.3:33209/ using document root <none> and password file <none>
I20260924 05:09:08.386920 4284 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:08.386991 4284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:08.387128 4284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:08.388250 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "daefdc1fe4e845de900ada5de7be8418"
format_stamp: "Formatted at 2026-09-24 05:09:08 on dist-test-slave-mr2f"
server_key: "bb60572bb602a5eda7832fd82fa1bf3d"
server_key_iv: "c32ba3768c8516b7fd7ced4e4116440a"
server_key_version: "encryptionkey@0"
I20260924 05:09:08.388621 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "daefdc1fe4e845de900ada5de7be8418"
format_stamp: "Formatted at 2026-09-24 05:09:08 on dist-test-slave-mr2f"
server_key: "bb60572bb602a5eda7832fd82fa1bf3d"
server_key_iv: "c32ba3768c8516b7fd7ced4e4116440a"
server_key_version: "encryptionkey@0"
I20260924 05:09:08.389925 4284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:08.390663 4299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:08.390862 4284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:08.390960 4284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "daefdc1fe4e845de900ada5de7be8418"
format_stamp: "Formatted at 2026-09-24 05:09:08 on dist-test-slave-mr2f"
server_key: "bb60572bb602a5eda7832fd82fa1bf3d"
server_key_iv: "c32ba3768c8516b7fd7ced4e4116440a"
server_key_version: "encryptionkey@0"
I20260924 05:09:08.391047 4284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:08.405876 4284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:08.408203 4284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:08.408377 4284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:08.408618 4284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:08.408932 4284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:08.408987 4284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:08.409065 4284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:08.409094 4284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:08.414870 4284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35237
I20260924 05:09:08.414911 4412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35237 every 8 connection(s)
I20260924 05:09:08.415215 4284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:08.419876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 4284
I20260924 05:09:08.419987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:08.420213 392 external_mini_cluster.cc:1428] Setting key 914a7d019c288fc78da905f2058b9517
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:08.424099 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:08.415961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34035 (local address 127.0.98.62:34377)
0924 05:09:08.416031 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:08.416032 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:08.416193 (+ 161us) server_negotiation.cc:407] Connection header received
0924 05:09:08.417098 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:08.417099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:08.417108 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:08.417167 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:08.417721 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.418212 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.419163 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.419349 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.419416 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:08.421889 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:08.421890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:08.421890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:08.421897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:08.423032 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.423384 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.423385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.423385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.423432 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.423606 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.423606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.423606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.423701 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:08.423752 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:08.423970 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 05:09:08.424029 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:09:08.424293 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:08.415922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34377 (local address 127.0.98.3:34035)
0924 05:09:08.416078 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:08.416119 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:08.417005 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:08.417229 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:08.417232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:08.417472 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:08.417652 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:08.417656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.418294 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.418295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:08.419089 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:08.419091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.419436 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.419437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:08.419527 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:08.420113 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:08.420121 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:08.421803 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:08.423105 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:08.423110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:08.423112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:08.423319 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:08.423472 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:08.423472 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:08.423472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:08.423553 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:08.423761 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:08.423765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:08.423871 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 05:09:08.423951 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 05:09:08.424062 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260924 05:09:08.424795 4413 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34377
I20260924 05:09:08.424906 4413 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:08.425081 4413 heartbeater.cc:511] Master 127.0.98.62:34377 requested a full tablet report, sending...
I20260924 05:09:08.425432 3954 ts_manager.cc:194] Registered new tserver with Master: daefdc1fe4e845de900ada5de7be8418 (127.0.98.3:35237)
I20260924 05:09:08.425810 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34035
I20260924 05:09:08.432137 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:09:08.454882 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:08.446955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.0.98.1:44741)
0924 05:09:08.447096 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:08.447101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:08.447116 (+ 15us) server_negotiation.cc:407] Connection header received
0924 05:09:08.447149 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:08.447151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:08.447164 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:08.447231 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:08.447826 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.448545 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.449290 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.449605 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.449662 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:08.452065 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:08.452071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:08.452072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:08.452078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:08.453644 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.453993 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.453996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.453996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.454050 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.454279 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.454279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.454280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.454399 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:08.454499 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:08.454636 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 05:09:08.454696 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:08.462841 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:08.456116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.0.98.62:34377)
0924 05:09:08.456175 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:08.456177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:08.456186 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:08.456262 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:08.456263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:08.456272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:08.456330 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:08.456804 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.457292 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.457960 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:08.458175 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:08.458260 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:08.460253 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:08.460255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:08.460255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:08.460262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:08.461491 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.461892 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.461893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.461893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.461955 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:08.462195 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:08.462196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:08.462196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:08.462353 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:08.462468 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:08.462623 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 05:09:08.462709 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 05:09:08.463092 3954 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43766
I20260924 05:09:08.463449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 4016
I20260924 05:09:08.468173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 4150
I20260924 05:09:08.473007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 4284
I20260924 05:09:08.477077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 3922
2026-09-24T05:09:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1809 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[4425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[4425](info): set up 2 sockets
Sep 24 05:09:08 dist-test-slave-mr2f krb5kdc[4425](info): commencing operation
krb5kdc: starting...
W20260924 05:09:09.502722 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:09 dist-test-slave-mr2f krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:09:09.613641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:09.690958 4438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:09.691170 4438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:09.691207 4438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:09.692312 4438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:09.692378 4438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:09.692402 4438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:09.692430 4438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:09.692461 4438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:09.693862 4438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:09.694118 4438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:09.694335 4438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:09.699242 4443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:09.699330 4438 server_base.cc:1034] running on GCE node
W20260924 05:09:09.699247 4444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:09.699265 4446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:09.699757 4438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:09.700035 4438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:09.701200 4438 hybrid_clock.cc:648] HybridClock initialized: now 1790226549701178 us; error 37 us; skew 500 ppm
Sep 24 05:09:09 dist-test-slave-mr2f krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:09.703666 4438 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:09:09.703994 4438 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:09.704041 4438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:09.704123 4438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:09.705083 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "fab94c869bdb48e996fb0002b301dd90"
format_stamp: "Formatted at 2026-09-24 05:09:09 on dist-test-slave-mr2f"
server_key: "41145283f873faf5338afa5ab923b254"
server_key_iv: "e08be34ed347fb6f2186897080547115"
server_key_version: "encryptionkey@0"
I20260924 05:09:09.705379 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "fab94c869bdb48e996fb0002b301dd90"
format_stamp: "Formatted at 2026-09-24 05:09:09 on dist-test-slave-mr2f"
server_key: "41145283f873faf5338afa5ab923b254"
server_key_iv: "e08be34ed347fb6f2186897080547115"
server_key_version: "encryptionkey@0"
I20260924 05:09:09.706545 4438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:09:09.707255 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:09.707432 4438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:09.707530 4438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "fab94c869bdb48e996fb0002b301dd90"
format_stamp: "Formatted at 2026-09-24 05:09:09 on dist-test-slave-mr2f"
server_key: "41145283f873faf5338afa5ab923b254"
server_key_iv: "e08be34ed347fb6f2186897080547115"
server_key_version: "encryptionkey@0"
I20260924 05:09:09.707607 4438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:09.729096 4438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:09.731463 4438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:09.731623 4438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:09.735443 4438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39343
I20260924 05:09:09.735481 4504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39343 every 8 connection(s)
I20260924 05:09:09.735807 4438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:09.736728 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:09.737702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 4438
I20260924 05:09:09.737823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:09.738022 392 external_mini_cluster.cc:1428] Setting key 6b3e78a9d259d0df19a0d0709309987e
I20260924 05:09:09.739676 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90: Bootstrap starting.
I20260924 05:09:09.740345 4505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:09.740665 4505 log.cc:824] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:09.741375 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90: No bootstrap required, opened a new log
I20260924 05:09:09.743541 4505 raft_consensus.cc:348] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab94c869bdb48e996fb0002b301dd90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39343 } }
I20260924 05:09:09.743667 4505 raft_consensus.cc:374] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:09.743690 4505 raft_consensus.cc:729] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab94c869bdb48e996fb0002b301dd90, State: Initialized, Role: FOLLOWER
I20260924 05:09:09.743831 4505 consensus_queue.cc:260] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab94c869bdb48e996fb0002b301dd90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39343 } }
I20260924 05:09:09.743906 4505 raft_consensus.cc:388] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:09.743928 4505 raft_consensus.cc:482] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:09.743995 4505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:09.744469 4505 raft_consensus.cc:504] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab94c869bdb48e996fb0002b301dd90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39343 } }
I20260924 05:09:09.744577 4505 leader_election.cc:302] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fab94c869bdb48e996fb0002b301dd90; no voters:
I20260924 05:09:09.744741 4505 leader_election.cc:288] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:09.744827 4510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:09.744961 4510 raft_consensus.cc:686] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [term 1 LEADER]: Becoming Leader. State: Replica: fab94c869bdb48e996fb0002b301dd90, State: Running, Role: LEADER
I20260924 05:09:09.745002 4505 sys_catalog.cc:564] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:09.745127 4510 consensus_queue.cc:237] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab94c869bdb48e996fb0002b301dd90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39343 } }
Sep 24 05:09:09 dist-test-slave-mr2f krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:09.745503 4512 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fab94c869bdb48e996fb0002b301dd90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab94c869bdb48e996fb0002b301dd90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39343 } } }
I20260924 05:09:09.745494 4510 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab94c869bdb48e996fb0002b301dd90. Latest consensus state: current_term: 1 leader_uuid: "fab94c869bdb48e996fb0002b301dd90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab94c869bdb48e996fb0002b301dd90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39343 } } }
I20260924 05:09:09.745586 4512 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:09.745596 4510 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:09.745812 4519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:09.746315 4519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:09.747915 4519 catalog_manager.cc:1347] Generated new cluster ID: e9617a00779743cf906f69088dcbf1ac
I20260924 05:09:09.747967 4519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:09.749519 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:09.738538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.0.98.62:39343)
0924 05:09:09.738734 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:09.738743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:09.738797 (+ 54us) server_negotiation.cc:407] Connection header received
0924 05:09:09.739385 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:09.739394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:09.739455 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:09.739616 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:09.740222 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:09.741608 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:09.742368 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:09.743084 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:09.743198 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:09.745879 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:09.745884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:09.745884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:09.745896 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:09.747963 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:09.748366 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:09.748368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:09.748369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:09.748462 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:09.748691 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:09.748692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:09.748693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:09.748960 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:09.749084 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:09.749224 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:09:09.749341 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260924 05:09:09.771417 4519 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:09.772157 4519 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:09.781507 4519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90: Generated new TSK 0
I20260924 05:09:09.781687 4519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:09:09.811529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39343
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:09.890503 4531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:09.890693 4531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:09.890717 4531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:09.891849 4531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:09.891891 4531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:09.891916 4531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:09.893417 4531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:09.893666 4531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:09.893903 4531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:09.898850 4536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:09.898872 4539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:09.898850 4537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:09.899000 4531 server_base.cc:1034] running on GCE node
I20260924 05:09:09.899235 4531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:09.899493 4531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:09.900650 4531 hybrid_clock.cc:648] HybridClock initialized: now 1790226549900638 us; error 29 us; skew 500 ppm
Sep 24 05:09:09 dist-test-slave-mr2f krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:09.903012 4531 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:09:09.903373 4531 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:09.903452 4531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:09.903599 4531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:09.904564 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "a368c203c9a440e0b1e620f02c542b0b"
format_stamp: "Formatted at 2026-09-24 05:09:09 on dist-test-slave-mr2f"
server_key: "da5e00d77512a06051888c99ed7baed8"
server_key_iv: "629a8a496d80438e3f1d5d8ad0898119"
server_key_version: "encryptionkey@0"
I20260924 05:09:09.904929 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a368c203c9a440e0b1e620f02c542b0b"
format_stamp: "Formatted at 2026-09-24 05:09:09 on dist-test-slave-mr2f"
server_key: "da5e00d77512a06051888c99ed7baed8"
server_key_iv: "629a8a496d80438e3f1d5d8ad0898119"
server_key_version: "encryptionkey@0"
I20260924 05:09:09.906172 4531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:09.906919 4545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:09.907116 4531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:09:09.907208 4531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "a368c203c9a440e0b1e620f02c542b0b"
format_stamp: "Formatted at 2026-09-24 05:09:09 on dist-test-slave-mr2f"
server_key: "da5e00d77512a06051888c99ed7baed8"
server_key_iv: "629a8a496d80438e3f1d5d8ad0898119"
server_key_version: "encryptionkey@0"
I20260924 05:09:09.907300 4531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:09.940165 4531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:09.942605 4531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:09.942785 4531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:09.943022 4531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:09.943336 4531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:09.943389 4531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:09.943431 4531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:09.943459 4531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:09.949385 4531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42353
I20260924 05:09:09.949437 4658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42353 every 8 connection(s)
I20260924 05:09:09.949723 4531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:09:09 dist-test-slave-mr2f krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:09.956243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 4531
I20260924 05:09:09.956370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:09.956555 392 external_mini_cluster.cc:1428] Setting key f0742afd5f388a4a7ba2a6b3c75184f2
I20260924 05:09:09.959057 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:09.950528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37579 (local address 127.0.98.62:39343)
0924 05:09:09.950576 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:09.950578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:09.950786 (+ 208us) server_negotiation.cc:407] Connection header received
0924 05:09:09.951332 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:09.951333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:09.951345 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:09.951406 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:09.952298 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:09.952809 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:09.953820 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:09.954049 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:09.954135 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:09.956159 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:09.956161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:09.956161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:09.956169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:09.957639 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:09.958040 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:09.958041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:09.958042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:09.958105 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:09.958409 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:09.958411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:09.958411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:09.958537 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:09.958597 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:09.958933 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 05:09:09.958999 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:09:09.959344 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:09.950496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39343 (local address 127.0.98.1:37579)
0924 05:09:09.950641 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:09.950681 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:09.951205 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:09.951476 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:09.951481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:09.951725 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:09.952190 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:09.952195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:09.952904 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:09.952905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:09.953701 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:09.953703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:09.954144 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:09.954145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:09.954277 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:09.954752 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:09.954777 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:09.956061 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:09.957753 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:09.957760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:09.957763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:09.957967 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:09.958170 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:09.958171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:09.958171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:09.958317 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:09.958608 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:09.958614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:09.958796 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 05:09:09.958913 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 05:09:09.959076 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260924 05:09:09.959774 4659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39343
I20260924 05:09:09.959861 4659 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:09.960016 4659 heartbeater.cc:511] Master 127.0.98.62:39343 requested a full tablet report, sending...
I20260924 05:09:09.960469 4469 ts_manager.cc:194] Registered new tserver with Master: a368c203c9a440e0b1e620f02c542b0b (127.0.98.1:42353)
I20260924 05:09:09.961064 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37579
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:09:09.979719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39343
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:10.057708 4664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:10.057940 4664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:10.057979 4664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:10.059149 4664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:10.059219 4664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:10.059271 4664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:10.060770 4664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:10.061021 4664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:10.061236 4664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:10.066164 4672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:10.066190 4664 server_base.cc:1034] running on GCE node
W20260924 05:09:10.066172 4669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:10.066205 4670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:10.066691 4664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:10.066973 4664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:10.068161 4664 hybrid_clock.cc:648] HybridClock initialized: now 1790226550068148 us; error 26 us; skew 500 ppm
Sep 24 05:09:10 dist-test-slave-mr2f krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:10.071161 4664 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:09:10.071440 4664 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:10.071482 4664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:10.071573 4664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:10.072546 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "5af2eb70445ebd33c7a55bdb52b79990"
server_key_iv: "4aa50421ab9bb80f1ae6be18bba55a35"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.072839 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "5af2eb70445ebd33c7a55bdb52b79990"
server_key_iv: "4aa50421ab9bb80f1ae6be18bba55a35"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.074074 4664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:10.074729 4678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.074906 4664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:10.075006 4664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "5af2eb70445ebd33c7a55bdb52b79990"
server_key_iv: "4aa50421ab9bb80f1ae6be18bba55a35"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.075102 4664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:10.089457 4664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:10.091252 4664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:10.091396 4664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:10.091607 4664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:10.091899 4664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:10.091953 4664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.091995 4664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:10.092022 4664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.097683 4664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43717
I20260924 05:09:10.097714 4791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43717 every 8 connection(s)
I20260924 05:09:10.097965 4664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:09:10 dist-test-slave-mr2f krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:10.104211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 4664
I20260924 05:09:10.104321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:10.104537 392 external_mini_cluster.cc:1428] Setting key 70d8c15a6e749719ed8f71f1789db3ba
I20260924 05:09:10.106685 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.098601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35653 (local address 127.0.98.62:39343)
0924 05:09:10.098642 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.098644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.098891 (+ 247us) server_negotiation.cc:407] Connection header received
0924 05:09:10.099318 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.099320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.099329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.099377 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:10.100210 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.100695 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.101622 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.101799 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.101849 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.103768 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:10.103769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:10.103770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:10.103776 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:10.105240 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:10.105711 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:10.105711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:10.105712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:10.105758 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:10.106018 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:10.106018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:10.106019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:10.106110 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:10.106159 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.106462 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.106562 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:09:10.106910 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.098591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39343 (local address 127.0.98.2:35653)
0924 05:09:10.098725 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.098755 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:10.099237 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:10.099426 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:10.099429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:10.099679 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:10.100138 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.100143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.100769 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.100769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.101541 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.101543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.101886 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.101887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.101978 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.102377 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:10.102384 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:10.103687 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:10.105321 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:10.105327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:10.105329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:10.105634 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:10.105821 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:10.105822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:10.105822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:10.105957 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:10.106172 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:10.106177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:10.106334 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 05:09:10.106444 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 05:09:10.106611 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":84,"threads_started":1}
I20260924 05:09:10.107486 4792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39343
I20260924 05:09:10.107597 4792 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:10.107755 4792 heartbeater.cc:511] Master 127.0.98.62:39343 requested a full tablet report, sending...
I20260924 05:09:10.108100 4469 ts_manager.cc:194] Registered new tserver with Master: fea3e67d2f2a4f109a65cc7a2ee08f89 (127.0.98.2:43717)
I20260924 05:09:10.108476 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35653
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:09:10.127688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39343
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:10.205873 4797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:10.206072 4797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:10.206096 4797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:10.207309 4797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:10.207350 4797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:10.207376 4797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:10.208876 4797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:10.209103 4797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:10.209323 4797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:10.214401 4802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:10.214416 4805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:10.214473 4803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:10.214520 4797 server_base.cc:1034] running on GCE node
I20260924 05:09:10.214886 4797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:10.215122 4797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:10.216261 4797 hybrid_clock.cc:648] HybridClock initialized: now 1790226550216246 us; error 33 us; skew 500 ppm
Sep 24 05:09:10 dist-test-slave-mr2f krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:10.218631 4797 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:09:10.219041 4797 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:10.219092 4797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:10.219249 4797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:10.220328 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "901014f66b784bf68474526c13197081"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "bc5b51e5069cd1b7976c8d9d4bfa58ea"
server_key_iv: "2967f89b0704a90cbb72137d026de1de"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.220676 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "901014f66b784bf68474526c13197081"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "bc5b51e5069cd1b7976c8d9d4bfa58ea"
server_key_iv: "2967f89b0704a90cbb72137d026de1de"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.221949 4797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:10.222647 4811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.222883 4797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:10.222980 4797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "901014f66b784bf68474526c13197081"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "bc5b51e5069cd1b7976c8d9d4bfa58ea"
server_key_iv: "2967f89b0704a90cbb72137d026de1de"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.223059 4797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:10.235924 4797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:10.238788 4797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:10.238946 4797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:10.239239 4797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:10.239625 4797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:10.239678 4797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.239709 4797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:10.239740 4797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.246071 4797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35243
I20260924 05:09:10.246145 4924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35243 every 8 connection(s)
I20260924 05:09:10.246392 4797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:10.251672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 4797
I20260924 05:09:10.251767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
Sep 24 05:09:10 dist-test-slave-mr2f krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:10.251950 392 external_mini_cluster.cc:1428] Setting key 96717bcf2cb6fb9dbd46a7b761d072c0
I20260924 05:09:10.254714 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.247074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43791 (local address 127.0.98.62:39343)
0924 05:09:10.247135 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.247137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.247335 (+ 198us) server_negotiation.cc:407] Connection header received
0924 05:09:10.247775 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.247776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.247786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.247848 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:10.248667 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.249137 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.250057 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.250238 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.250304 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.252506 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:10.252508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:10.252508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:10.252515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:10.253663 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:10.253998 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:10.253998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:10.253999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:10.254046 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:10.254240 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:10.254240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:10.254241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:10.254340 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:10.254388 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.254597 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.254654 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 05:09:10.254906 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.247050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39343 (local address 127.0.98.3:43791)
0924 05:09:10.247194 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.247251 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:10.247699 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:10.247900 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:10.247904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:10.248137 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:10.248596 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.248601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.249214 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.249215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.249987 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.249988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.250326 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.250326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.250414 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.250848 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:10.250856 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:10.252416 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:10.253726 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:10.253730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:10.253732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:10.253933 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:10.254099 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:10.254100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:10.254100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:10.254188 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:10.254390 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:10.254394 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:10.254494 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 05:09:10.254569 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:09:10.254683 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260924 05:09:10.255378 4925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39343
I20260924 05:09:10.255479 4925 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:10.255632 4925 heartbeater.cc:511] Master 127.0.98.62:39343 requested a full tablet report, sending...
I20260924 05:09:10.255962 4469 ts_manager.cc:194] Registered new tserver with Master: 901014f66b784bf68474526c13197081 (127.0.98.3:35243)
I20260924 05:09:10.256320 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43791
I20260924 05:09:10.263360 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:09:10.269719 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.264444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.0.98.62:39343)
0924 05:09:10.264486 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.264487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.264549 (+ 62us) server_negotiation.cc:407] Connection header received
0924 05:09:10.264605 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.264605 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.264614 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.264660 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:10.265163 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.265602 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.266263 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.266428 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.266475 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.267551 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:10.267552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:10.267552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:10.267558 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:10.268885 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:10.269158 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:10.269159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:10.269159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:10.269205 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:10.269349 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:10.269350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:10.269350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:10.269460 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:10.269531 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.269608 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.269663 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:10.271217 4469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:10.272610 4469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:10.278680 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.274634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.0.1:60776)
0924 05:09:10.274969 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.274980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:10.275060 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:10.275397 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:10.275397 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:10.275399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:10.275439 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.275441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.276369 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.276370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.277441 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.277443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.278375 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.278376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.278452 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.278465 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:09:10.278548 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:09:10.278600 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":75,"threads_started":1}
I20260924 05:09:10.278861 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.274858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35243 (local address 127.0.0.1:56040)
0924 05:09:10.274990 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.275000 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:10.275044 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:10.275370 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:10.275373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:10.275375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:10.275439 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.275444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.276793 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.276794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.277816 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.277818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.278602 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.278603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.278660 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.278663 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:10.278717 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 05:09:10.278754 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":46,"thread_start_us":42,"threads_started":1}
I20260924 05:09:10.279117 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.274718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.0.98.2:43717)
0924 05:09:10.274931 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.274936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.275036 (+ 100us) server_negotiation.cc:407] Connection header received
0924 05:09:10.275171 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.275175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.275192 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.275281 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:10.275512 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.276270 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.277556 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.278292 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.278365 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.278416 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.279013 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.279053 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260924 05:09:10.279237 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.275032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.0.98.3:35243)
0924 05:09:10.275172 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.275176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.275186 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:10.275222 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.275223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.275234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.275518 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:10.275605 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.276674 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.277894 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.278489 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.278555 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.278607 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.279121 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.279163 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260924 05:09:10.279803 4726 tablet_service.cc:1467] Processing CreateTablet for tablet db3aa0ac9c8f4b428109481ef36b3142 (DEFAULT_TABLE table=test-table [id=8e6ee1e4093c480796bfdd73b04b4405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:10.279803 4859 tablet_service.cc:1467] Processing CreateTablet for tablet db3aa0ac9c8f4b428109481ef36b3142 (DEFAULT_TABLE table=test-table [id=8e6ee1e4093c480796bfdd73b04b4405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:10.280139 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.274630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42353 (local address 127.0.0.1:53370)
0924 05:09:10.274838 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.274877 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:10.274942 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:10.275557 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:10.275558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:10.275559 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:10.275606 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.275607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.276834 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.276835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.278906 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.278909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.279889 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.279890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.279945 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.279947 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:09:10.280014 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 05:09:10.280067 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":65,"threads_started":1}
I20260924 05:09:10.280155 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.274787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.0.98.1:42353)
0924 05:09:10.275216 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.275222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.275232 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:10.275268 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.275270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.275282 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.275354 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:10.275676 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.276736 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.278990 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.279747 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.279809 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.279861 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.280025 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.280083 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":74,"threads_started":1}
I20260924 05:09:10.280154 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3aa0ac9c8f4b428109481ef36b3142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:10.280154 4859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3aa0ac9c8f4b428109481ef36b3142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:10.283442 4939 tablet_bootstrap.cc:492] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: Bootstrap starting.
I20260924 05:09:10.283375 4593 tablet_service.cc:1467] Processing CreateTablet for tablet db3aa0ac9c8f4b428109481ef36b3142 (DEFAULT_TABLE table=test-table [id=8e6ee1e4093c480796bfdd73b04b4405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:10.283701 4593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3aa0ac9c8f4b428109481ef36b3142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:10.283799 4940 tablet_bootstrap.cc:492] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: Bootstrap starting.
I20260924 05:09:10.284042 4939 tablet_bootstrap.cc:654] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:10.284355 4940 tablet_bootstrap.cc:654] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:10.284608 4939 log.cc:824] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:10.284919 4940 log.cc:824] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:10.285333 4939 tablet_bootstrap.cc:492] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: No bootstrap required, opened a new log
I20260924 05:09:10.285388 4939 ts_tablet_manager.cc:1397] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:10.285751 4940 tablet_bootstrap.cc:492] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: No bootstrap required, opened a new log
I20260924 05:09:10.285815 4940 ts_tablet_manager.cc:1397] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:10.286725 4943 tablet_bootstrap.cc:492] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b: Bootstrap starting.
I20260924 05:09:10.287392 4943 tablet_bootstrap.cc:654] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:10.287370 4939 raft_consensus.cc:348] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 05:09:10.287541 4939 raft_consensus.cc:374] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:10.287581 4939 raft_consensus.cc:729] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901014f66b784bf68474526c13197081, State: Initialized, Role: FOLLOWER
I20260924 05:09:10.287707 4939 consensus_queue.cc:260] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 05:09:10.287758 4940 raft_consensus.cc:348] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 05:09:10.287911 4940 raft_consensus.cc:374] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:10.287945 4940 raft_consensus.cc:729] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea3e67d2f2a4f109a65cc7a2ee08f89, State: Initialized, Role: FOLLOWER
I20260924 05:09:10.287946 4943 log.cc:824] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:10.287992 4939 ts_tablet_manager.cc:1428] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:09:10.288113 4925 heartbeater.cc:503] Master 127.0.98.62:39343 was elected leader, sending a full tablet report...
I20260924 05:09:10.288082 4940 consensus_queue.cc:260] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 05:09:10.288364 4792 heartbeater.cc:503] Master 127.0.98.62:39343 was elected leader, sending a full tablet report...
I20260924 05:09:10.288353 4940 ts_tablet_manager.cc:1428] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:09:10.288662 4943 tablet_bootstrap.cc:492] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b: No bootstrap required, opened a new log
I20260924 05:09:10.288717 4943 ts_tablet_manager.cc:1397] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:10.290192 4943 raft_consensus.cc:348] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 05:09:10.290328 4943 raft_consensus.cc:374] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:10.290373 4943 raft_consensus.cc:729] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a368c203c9a440e0b1e620f02c542b0b, State: Initialized, Role: FOLLOWER
I20260924 05:09:10.290477 4943 consensus_queue.cc:260] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 05:09:10.290742 4943 ts_tablet_manager.cc:1428] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:09:10.290748 4659 heartbeater.cc:503] Master 127.0.98.62:39343 was elected leader, sending a full tablet report...
W20260924 05:09:10.348538 4793 tablet.cc:2369] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:09:10.450460 4660 tablet.cc:2369] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:09:10.451346 4947 raft_consensus.cc:482] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:10.451452 4947 raft_consensus.cc:504] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 05:09:10.451774 4947 leader_election.cc:288] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fea3e67d2f2a4f109a65cc7a2ee08f89 (127.0.98.2:43717), 901014f66b784bf68474526c13197081 (127.0.98.3:35243)
I20260924 05:09:10.454618 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.452043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60123 (local address 127.0.98.2:43717)
0924 05:09:10.452090 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.452092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.452263 (+ 171us) server_negotiation.cc:407] Connection header received
0924 05:09:10.452292 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.452293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.452304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.452362 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:10.452543 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.452966 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.453910 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.454262 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.454340 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.454350 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.454520 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.454559 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:09:10.454610 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.451951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35243 (local address 127.0.98.1:33327)
0924 05:09:10.452010 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.452022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:10.452073 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:10.452288 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:10.452289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:10.452291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:10.452369 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.452371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.453037 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.453038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.453812 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.453814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.454349 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.454350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.454424 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.454427 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:10.454468 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 05:09:10.454525 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":22}
I20260924 05:09:10.454623 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.451954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.98.1:60123)
0924 05:09:10.452161 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.452171 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:10.452234 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:10.452377 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:10.452378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:10.452379 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:10.452469 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.452470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.453082 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.453083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.453813 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:10.453815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.454359 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.454360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:10.454429 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.454432 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:10.454498 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 05:09:10.454533 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":75,"thread_start_us":72,"threads_started":1}
I20260924 05:09:10.454901 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.452094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33327 (local address 127.0.98.3:35243)
0924 05:09:10.452134 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.452135 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.452144 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:10.452174 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.452175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.452199 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.452258 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:10.452447 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.452948 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.453908 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.454265 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.454351 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.454360 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.454493 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.454530 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:09:10.455212 4746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3aa0ac9c8f4b428109481ef36b3142" candidate_uuid: "a368c203c9a440e0b1e620f02c542b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" is_pre_election: true
I20260924 05:09:10.455251 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3aa0ac9c8f4b428109481ef36b3142" candidate_uuid: "a368c203c9a440e0b1e620f02c542b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901014f66b784bf68474526c13197081" is_pre_election: true
I20260924 05:09:10.455364 4746 raft_consensus.cc:2454] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a368c203c9a440e0b1e620f02c542b0b in term 0.
I20260924 05:09:10.455422 4879 raft_consensus.cc:2454] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a368c203c9a440e0b1e620f02c542b0b in term 0.
I20260924 05:09:10.455614 4546 leader_election.cc:302] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a368c203c9a440e0b1e620f02c542b0b, fea3e67d2f2a4f109a65cc7a2ee08f89; no voters:
I20260924 05:09:10.455756 4947 raft_consensus.cc:2781] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:10.455816 4947 raft_consensus.cc:482] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:10.455850 4947 raft_consensus.cc:3037] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:10.456488 4947 raft_consensus.cc:504] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 05:09:10.456630 4947 leader_election.cc:288] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [CANDIDATE]: Term 1 election: Requested vote from peers fea3e67d2f2a4f109a65cc7a2ee08f89 (127.0.98.2:43717), 901014f66b784bf68474526c13197081 (127.0.98.3:35243)
I20260924 05:09:10.456771 4746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3aa0ac9c8f4b428109481ef36b3142" candidate_uuid: "a368c203c9a440e0b1e620f02c542b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89"
I20260924 05:09:10.456796 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3aa0ac9c8f4b428109481ef36b3142" candidate_uuid: "a368c203c9a440e0b1e620f02c542b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901014f66b784bf68474526c13197081"
I20260924 05:09:10.456851 4746 raft_consensus.cc:3037] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:10.456856 4879 raft_consensus.cc:3037] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:10.457708 4879 raft_consensus.cc:2454] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a368c203c9a440e0b1e620f02c542b0b in term 1.
I20260924 05:09:10.457710 4746 raft_consensus.cc:2454] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a368c203c9a440e0b1e620f02c542b0b in term 1.
I20260924 05:09:10.457885 4546 leader_election.cc:302] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a368c203c9a440e0b1e620f02c542b0b, fea3e67d2f2a4f109a65cc7a2ee08f89; no voters:
I20260924 05:09:10.458006 4947 raft_consensus.cc:2781] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:10.458153 4947 raft_consensus.cc:686] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [term 1 LEADER]: Becoming Leader. State: Replica: a368c203c9a440e0b1e620f02c542b0b, State: Running, Role: LEADER
I20260924 05:09:10.458253 4947 consensus_queue.cc:237] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } }
I20260924 05:09:10.459098 4469 catalog_manager.cc:5495] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b reported cstate change: term changed from 0 to 1, leader changed from <none> to a368c203c9a440e0b1e620f02c542b0b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a368c203c9a440e0b1e620f02c542b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a368c203c9a440e0b1e620f02c542b0b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 } health_report { overall_health: UNKNOWN } } }
W20260924 05:09:10.496959 4926 tablet.cc:2369] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:09:10.503912 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.501753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.0.98.1:42353)
0924 05:09:10.501809 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.501811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.501821 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:10.501870 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.501871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.501891 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.501971 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:10.502088 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.502546 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.503270 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:10.503455 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:10.503510 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:10.503635 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:10.503730 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:10.503792 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.503822 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.503839 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:09:10.505743 4746 raft_consensus.cc:1261] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [term 1 FOLLOWER]: Refusing update from remote peer a368c203c9a440e0b1e620f02c542b0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:10.505743 4879 raft_consensus.cc:1261] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [term 1 FOLLOWER]: Refusing update from remote peer a368c203c9a440e0b1e620f02c542b0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:10.506140 4949 consensus_queue.cc:1035] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "901014f66b784bf68474526c13197081" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:10.506332 4947 consensus_queue.cc:1035] T db3aa0ac9c8f4b428109481ef36b3142 P a368c203c9a440e0b1e620f02c542b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea3e67d2f2a4f109a65cc7a2ee08f89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:10.509040 4952 mvcc.cc:205] Tried to move back new op lower bound from 7332767950868492288 to 7332767950677860352. Current Snapshot: MvccSnapshot[applied={T|T < 7332767950868492288}]
I20260924 05:09:10.509146 4954 mvcc.cc:205] Tried to move back new op lower bound from 7332767950868492288 to 7332767950677860352. Current Snapshot: MvccSnapshot[applied={T|T < 7332767950868492288 or (T in {7332767950868492288})}]
I20260924 05:09:10.511103 4468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:10.511209 4468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:10.511962 4468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fab94c869bdb48e996fb0002b301dd90: Sending DeleteTablet for 3 replicas of tablet db3aa0ac9c8f4b428109481ef36b3142
I20260924 05:09:10.512267 4593 tablet_service.cc:1514] Processing DeleteTablet for tablet db3aa0ac9c8f4b428109481ef36b3142 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53370
I20260924 05:09:10.512317 4859 tablet_service.cc:1514] Processing DeleteTablet for tablet db3aa0ac9c8f4b428109481ef36b3142 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56040
I20260924 05:09:10.512302 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet db3aa0ac9c8f4b428109481ef36b3142 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60776
I20260924 05:09:10.512511 4962 tablet_replica.cc:330] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: stopping tablet replica
I20260924 05:09:10.512609 4962 raft_consensus.cc:2229] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:10.512669 4963 tablet_replica.cc:330] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: stopping tablet replica
I20260924 05:09:10.512787 4963 raft_consensus.cc:2229] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:10.512794 4962 raft_consensus.cc:2258] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:10.512843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 4531
I20260924 05:09:10.513558 4963 raft_consensus.cc:2258] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:10.513972 4963 ts_tablet_manager.cc:1905] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:10.514432 4962 ts_tablet_manager.cc:1905] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:10.515408 4963 ts_tablet_manager.cc:1918] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:10.515455 4963 log.cc:1196] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/db3aa0ac9c8f4b428109481ef36b3142
I20260924 05:09:10.515666 4963 ts_tablet_manager.cc:1939] T db3aa0ac9c8f4b428109481ef36b3142 P fea3e67d2f2a4f109a65cc7a2ee08f89: Deleting consensus metadata
I20260924 05:09:10.516045 4453 catalog_manager.cc:4841] TS fea3e67d2f2a4f109a65cc7a2ee08f89 (127.0.98.2:43717): tablet db3aa0ac9c8f4b428109481ef36b3142 (table test-table [id=8e6ee1e4093c480796bfdd73b04b4405]) successfully deleted
I20260924 05:09:10.516188 4962 ts_tablet_manager.cc:1918] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:10.516260 4962 log.cc:1196] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/db3aa0ac9c8f4b428109481ef36b3142
I20260924 05:09:10.516499 4962 ts_tablet_manager.cc:1939] T db3aa0ac9c8f4b428109481ef36b3142 P 901014f66b784bf68474526c13197081: Deleting consensus metadata
I20260924 05:09:10.516894 4456 catalog_manager.cc:4841] TS 901014f66b784bf68474526c13197081 (127.0.98.3:35243): tablet db3aa0ac9c8f4b428109481ef36b3142 (table test-table [id=8e6ee1e4093c480796bfdd73b04b4405]) successfully deleted
W20260924 05:09:10.519819 4456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42353 (error 108)
I20260924 05:09:10.520174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 4664
I20260924 05:09:10.520284 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.520157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42353 (local address 127.0.0.1:53388)
0924 05:09:10.520194 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.520232 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260924 05:09:10.520864 4456 catalog_manager.cc:4584] TS a368c203c9a440e0b1e620f02c542b0b (127.0.98.1:42353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db3aa0ac9c8f4b428109481ef36b3142: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42353: connect: Connection refused (error 111)
I20260924 05:09:10.525919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 4797
I20260924 05:09:10.530323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 4438
2026-09-24T05:09:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2053 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T05:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:10Z Disabled control of system clock
I20260924 05:09:10.557189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46411
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 05:09:10.636808 4968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:10.637106 4968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:10.637143 4968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:10.638283 4968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:10.638346 4968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:10.638370 4968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:10.638399 4968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:10.638425 4968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:10.639846 4968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:10.640115 4968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:10.640383 4968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:10.645018 4974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:10.645044 4976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:10.645074 4973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:10.645452 4968 server_base.cc:1034] running on GCE node
I20260924 05:09:10.645685 4968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:10.645980 4968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:10.647126 4968 hybrid_clock.cc:648] HybridClock initialized: now 1790226550647123 us; error 40 us; skew 500 ppm
I20260924 05:09:10.648291 4968 webserver.cc:460] Webserver started at http://127.0.98.62:44075/ using document root <none> and password file <none>
I20260924 05:09:10.648540 4968 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:10.648586 4968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:10.648731 4968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:10.649765 4968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "ff808a6d84ce474c9f53d90ed3751af5"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "e683f2d73cba5d1aa103794cc583cf5d"
server_key_iv: "f6ea431052c97db9dd3466a4574c6397"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.650113 4968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff808a6d84ce474c9f53d90ed3751af5"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "e683f2d73cba5d1aa103794cc583cf5d"
server_key_iv: "f6ea431052c97db9dd3466a4574c6397"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.651501 4968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:10.652245 4982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.652427 4968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:10.652534 4968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "ff808a6d84ce474c9f53d90ed3751af5"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "e683f2d73cba5d1aa103794cc583cf5d"
server_key_iv: "f6ea431052c97db9dd3466a4574c6397"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.652614 4968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:10.655825 4968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:10.656078 4968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:10.656216 4968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:10.659806 4968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46411
I20260924 05:09:10.659834 5034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46411 every 8 connection(s)
I20260924 05:09:10.660213 4968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:10.660862 5035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:10.661262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 4968
I20260924 05:09:10.661374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:10.661536 392 external_mini_cluster.cc:1428] Setting key cca9d8fd169077308b295366efa9e577
I20260924 05:09:10.664201 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.662070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.0.98.62:46411)
0924 05:09:10.662293 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.662303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.662336 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:09:10.662930 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.662939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.663007 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.663161 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:10.663748 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:10.663752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:10.663752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:10.663764 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:10.663812 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:10.663826 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:10.663898 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.663957 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.664014 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
I20260924 05:09:10.664688 5035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5: Bootstrap starting.
I20260924 05:09:10.665272 5035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:10.665537 5035 log.cc:824] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:10.666180 5035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5: No bootstrap required, opened a new log
I20260924 05:09:10.667515 5035 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff808a6d84ce474c9f53d90ed3751af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46411 } }
I20260924 05:09:10.667666 5035 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:10.667713 5035 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff808a6d84ce474c9f53d90ed3751af5, State: Initialized, Role: FOLLOWER
I20260924 05:09:10.667832 5035 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff808a6d84ce474c9f53d90ed3751af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46411 } }
I20260924 05:09:10.667922 5035 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:10.667965 5035 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:10.668016 5035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:10.668514 5035 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff808a6d84ce474c9f53d90ed3751af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46411 } }
I20260924 05:09:10.668627 5035 leader_election.cc:302] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff808a6d84ce474c9f53d90ed3751af5; no voters:
I20260924 05:09:10.668802 5035 leader_election.cc:288] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:10.669054 5035 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:10.669078 5040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:10.669196 5040 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [term 1 LEADER]: Becoming Leader. State: Replica: ff808a6d84ce474c9f53d90ed3751af5, State: Running, Role: LEADER
I20260924 05:09:10.669320 5040 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff808a6d84ce474c9f53d90ed3751af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46411 } }
I20260924 05:09:10.669749 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff808a6d84ce474c9f53d90ed3751af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff808a6d84ce474c9f53d90ed3751af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46411 } } }
I20260924 05:09:10.669840 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:10.669737 5040 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff808a6d84ce474c9f53d90ed3751af5. Latest consensus state: current_term: 1 leader_uuid: "ff808a6d84ce474c9f53d90ed3751af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff808a6d84ce474c9f53d90ed3751af5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46411 } } }
I20260924 05:09:10.669907 5040 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:10.670069 5049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:10.670503 5049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:10.671944 5049 catalog_manager.cc:1347] Generated new cluster ID: 30466f516e9c4f118c3573e2fd8f79a8
I20260924 05:09:10.672013 5049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:10.679755 5049 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:10.679931 5049 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:10.688475 5049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5: Generated new TSK 0
I20260924 05:09:10.688657 5049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:09:10.689698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46411
--builtin_ntp_servers=127.0.98.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 05:09:10.769028 5059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:10.769263 5059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:10.769320 5059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:10.770648 5059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:10.770704 5059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:10.770749 5059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:10.772168 5059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:10.772416 5059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:10.772688 5059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:10.777647 5067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:10.777666 5064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:10.777653 5065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:10.778090 5059 server_base.cc:1034] running on GCE node
I20260924 05:09:10.778328 5059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:10.778582 5059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:10.779769 5059 hybrid_clock.cc:648] HybridClock initialized: now 1790226550779744 us; error 44 us; skew 500 ppm
I20260924 05:09:10.781002 5059 webserver.cc:460] Webserver started at http://127.0.98.1:41715/ using document root <none> and password file <none>
I20260924 05:09:10.781239 5059 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:10.781314 5059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:10.781481 5059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:10.782519 5059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "2d837a028f034a379375cb532cc783aa"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "e3a109b92f5013326006abf6a66e8802"
server_key_iv: "fd8c4def2cd342da096b7dc5ba804aad"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.782910 5059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2d837a028f034a379375cb532cc783aa"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "e3a109b92f5013326006abf6a66e8802"
server_key_iv: "fd8c4def2cd342da096b7dc5ba804aad"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.784231 5059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:10.785010 5073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.785197 5059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:10.785293 5059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "2d837a028f034a379375cb532cc783aa"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "e3a109b92f5013326006abf6a66e8802"
server_key_iv: "fd8c4def2cd342da096b7dc5ba804aad"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.785379 5059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:10.788661 5059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:10.788928 5059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:10.789088 5059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:10.789296 5059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:10.789590 5059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:10.789644 5059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.789680 5059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:10.789706 5059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.795331 5059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37649
I20260924 05:09:10.795398 5186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37649 every 8 connection(s)
I20260924 05:09:10.795714 5059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:09:10.798584 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.796426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46053 (local address 127.0.98.62:46411)
0924 05:09:10.796486 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.796488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.796736 (+ 248us) server_negotiation.cc:407] Connection header received
0924 05:09:10.797218 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.797219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.797225 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.797281 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:10.798333 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:10.798335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:10.798335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:10.798340 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:10.798354 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:10.798358 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:10.798404 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.798501 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.798516 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:09:10.798818 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.796433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46411 (local address 127.0.98.1:46053)
0924 05:09:10.796610 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.796642 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:10.797111 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:10.797355 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:10.797359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:10.797366 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:10.798188 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:10.798188 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:10.798214 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:10.798238 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:10.798412 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:10.798417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:10.798420 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:10.798509 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 05:09:10.798576 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260924 05:09:10.799297 5187 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46411
I20260924 05:09:10.799379 5187 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:10.799480 5187 heartbeater.cc:511] Master 127.0.98.62:46411 requested a full tablet report, sending...
I20260924 05:09:10.799954 4999 ts_manager.cc:194] Registered new tserver with Master: 2d837a028f034a379375cb532cc783aa (127.0.98.1:37649)
I20260924 05:09:10.804371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 5059
I20260924 05:09:10.804486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:10.804680 392 external_mini_cluster.cc:1428] Setting key c98b2393057a39184a2c81dc8c44a228
I20260924 05:09:10.805503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46411
--builtin_ntp_servers=127.0.98.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 05:09:10.884362 5190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:10.884555 5190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:10.884577 5190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:10.885728 5190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:10.885768 5190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:10.885794 5190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:10.887135 5190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:10.887387 5190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:10.887729 5190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:10.892619 5195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:10.892794 5190 server_base.cc:1034] running on GCE node
W20260924 05:09:10.892614 5198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:10.892614 5196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:10.893080 5190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:10.893324 5190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:10.894479 5190 hybrid_clock.cc:648] HybridClock initialized: now 1790226550894469 us; error 25 us; skew 500 ppm
I20260924 05:09:10.895787 5190 webserver.cc:460] Webserver started at http://127.0.98.2:37781/ using document root <none> and password file <none>
I20260924 05:09:10.896059 5190 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:10.896113 5190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:10.896287 5190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:10.897369 5190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4ca665fadcf423d9d5aa31adcfe8534"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "68c82843f450808eefa23b897fbffc71"
server_key_iv: "3ee1abb4f3c9e6af13ec760d98234a81"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.897847 5190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4ca665fadcf423d9d5aa31adcfe8534"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "68c82843f450808eefa23b897fbffc71"
server_key_iv: "3ee1abb4f3c9e6af13ec760d98234a81"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.899278 5190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:10.900135 5204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.900343 5190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:10.900444 5190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "e4ca665fadcf423d9d5aa31adcfe8534"
format_stamp: "Formatted at 2026-09-24 05:09:10 on dist-test-slave-mr2f"
server_key: "68c82843f450808eefa23b897fbffc71"
server_key_iv: "3ee1abb4f3c9e6af13ec760d98234a81"
server_key_version: "encryptionkey@0"
I20260924 05:09:10.900523 5190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:10.904136 5190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:10.904422 5190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:10.904577 5190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:10.904786 5190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:10.905102 5190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:10.905138 5190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.905197 5190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:10.905229 5190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:10.911188 5190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45435
I20260924 05:09:10.911247 5317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45435 every 8 connection(s)
I20260924 05:09:10.911561 5190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:10.914428 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.912269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48455 (local address 127.0.98.62:46411)
0924 05:09:10.912331 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:10.912334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:10.912573 (+ 239us) server_negotiation.cc:407] Connection header received
0924 05:09:10.913000 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:10.913001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:10.913005 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:10.913080 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:10.914209 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:10.914210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:10.914210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:10.914214 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:10.914229 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:10.914233 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:10.914281 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:10.914351 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 05:09:10.914372 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:09:10.914584 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:10.912252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46411 (local address 127.0.98.2:48455)
0924 05:09:10.912437 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:10.912476 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:10.912909 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:10.913128 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:10.913131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:10.913138 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:10.914071 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:10.914071 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:10.914097 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:10.914124 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:10.914288 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:10.914290 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:10.914292 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:09:10.914343 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:09:10.914387 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260924 05:09:10.915107 5318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46411
I20260924 05:09:10.915189 5318 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:10.915347 5318 heartbeater.cc:511] Master 127.0.98.62:46411 requested a full tablet report, sending...
I20260924 05:09:10.915707 4999 ts_manager.cc:194] Registered new tserver with Master: e4ca665fadcf423d9d5aa31adcfe8534 (127.0.98.2:45435)
I20260924 05:09:10.919915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 5190
I20260924 05:09:10.920020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:10.920213 392 external_mini_cluster.cc:1428] Setting key 42e20269de7aaaa4c58811a35595d65b
I20260924 05:09:10.921101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46411
--builtin_ntp_servers=127.0.98.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 05:09:11.001165 5321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:11.001369 5321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:11.001391 5321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:11.002517 5321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:11.002553 5321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:11.002579 5321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:11.004056 5321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:11.004303 5321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:11.004622 5321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:11.009536 5329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:11.009547 5327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:11.009615 5321 server_base.cc:1034] running on GCE node
W20260924 05:09:11.009542 5326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:11.009968 5321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:11.010159 5321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:11.011286 5321 hybrid_clock.cc:648] HybridClock initialized: now 1790226551011264 us; error 38 us; skew 500 ppm
I20260924 05:09:11.012394 5321 webserver.cc:460] Webserver started at http://127.0.98.3:37983/ using document root <none> and password file <none>
I20260924 05:09:11.012619 5321 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:11.012669 5321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:11.012836 5321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:11.013862 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "2d83e186d2e34cb6b51266dd5b3cac62"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "956bc0846b02a60c7dd6f7f535d1db89"
server_key_iv: "21b4bc093b4dbea9378f0e9927240209"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.014214 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2d83e186d2e34cb6b51266dd5b3cac62"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "956bc0846b02a60c7dd6f7f535d1db89"
server_key_iv: "21b4bc093b4dbea9378f0e9927240209"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.015576 5321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:11.016251 5335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:11.016428 5321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:11.016511 5321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "2d83e186d2e34cb6b51266dd5b3cac62"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "956bc0846b02a60c7dd6f7f535d1db89"
server_key_iv: "21b4bc093b4dbea9378f0e9927240209"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.016585 5321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:11.019868 5321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:11.020114 5321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:11.020257 5321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:11.020450 5321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:11.020737 5321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:11.020768 5321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:11.020821 5321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:11.020848 5321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:11.026425 5321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44251
I20260924 05:09:11.026467 5448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44251 every 8 connection(s)
I20260924 05:09:11.026819 5321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:11.029733 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.027514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44559 (local address 127.0.98.62:46411)
0924 05:09:11.027557 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.027559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.027808 (+ 249us) server_negotiation.cc:407] Connection header received
0924 05:09:11.028306 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.028307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.028314 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.028371 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.029453 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.029454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.029455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.029460 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.029474 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.029478 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.029554 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.029649 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.029664 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:09:11.029932 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.027509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46411 (local address 127.0.98.3:44559)
0924 05:09:11.027692 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.027723 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.028191 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.028436 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.028440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.028449 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:11.029307 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:11.029307 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:11.029333 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:11.029358 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:11.029555 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:11.029561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:11.029564 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:11.029659 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 05:09:11.029717 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260924 05:09:11.030388 5449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46411
I20260924 05:09:11.030462 5449 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:11.030623 5449 heartbeater.cc:511] Master 127.0.98.62:46411 requested a full tablet report, sending...
I20260924 05:09:11.030994 4999 ts_manager.cc:194] Registered new tserver with Master: 2d83e186d2e34cb6b51266dd5b3cac62 (127.0.98.3:44251)
I20260924 05:09:11.035528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 5321
I20260924 05:09:11.035625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:11.035771 392 external_mini_cluster.cc:1428] Setting key bf41eaae41288c2657fcdddf1ffbf1a3
I20260924 05:09:11.036551 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:09:11.038820 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.037760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.0.98.62:46411)
0924 05:09:11.037841 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.037842 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.037971 (+ 129us) server_negotiation.cc:407] Connection header received
0924 05:09:11.038020 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.038021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.038025 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.038062 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.038625 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.038626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.038626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.038629 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.038637 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.038640 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.038676 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.038709 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.038718 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 05:09:11.040275 4999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:11.041786 4999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:11.045439 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.043934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44251 (local address 127.0.0.1:56364)
0924 05:09:11.044237 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.044246 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.044281 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.044500 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.044506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.044514 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:11.044995 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:11.044995 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:11.045011 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:11.045031 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:11.045271 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:11.045276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:11.045279 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:11.045341 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 05:09:11.045354 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":98,"threads_started":1}
I20260924 05:09:11.045565 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.043988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.0.98.1:37649)
0924 05:09:11.044164 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.044169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.044180 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:09:11.044248 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.044253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.044265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.044343 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.045227 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.045236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.045237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.045241 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.045275 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.045310 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.045365 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.045467 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.045482 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260924 05:09:11.045639 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.044142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.0.98.3:44251)
0924 05:09:11.044378 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.044380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.044388 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:11.044406 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.044408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.044414 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.044476 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.045135 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.045144 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.045144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.045148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.045181 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.045205 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.045264 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.045536 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.045550 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260924 05:09:11.045704 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.043906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37649 (local address 127.0.0.1:50316)
0924 05:09:11.044081 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.044116 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.044171 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.044656 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.044657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.044659 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:11.045134 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:11.045134 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:11.045140 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:11.045148 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:11.045388 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:11.045388 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:11.045389 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:09:11.045429 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:09:11.045439 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":53,"threads_started":1}
I20260924 05:09:11.045835 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.044151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.0.98.2:45435)
0924 05:09:11.044316 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.044320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.044827 (+ 507us) server_negotiation.cc:407] Connection header received
0924 05:09:11.044861 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.044863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.044872 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.044977 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.045485 (+ 508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.045493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.045493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.045498 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.045530 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.045555 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.045614 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.045739 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.045753 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260924 05:09:11.046370 5121 tablet_service.cc:1467] Processing CreateTablet for tablet 5243908ebb2b498d949e271fce13024a (DEFAULT_TABLE table=test-table [id=151f70d3535d42b99e4182623da4e4a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:11.046440 5383 tablet_service.cc:1467] Processing CreateTablet for tablet 5243908ebb2b498d949e271fce13024a (DEFAULT_TABLE table=test-table [id=151f70d3535d42b99e4182623da4e4a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:11.046700 5121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5243908ebb2b498d949e271fce13024a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:11.046799 5383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5243908ebb2b498d949e271fce13024a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:11.049273 5464 tablet_bootstrap.cc:492] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: Bootstrap starting.
I20260924 05:09:11.049361 5463 tablet_bootstrap.cc:492] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa: Bootstrap starting.
I20260924 05:09:11.049467 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.043941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45435 (local address 127.0.0.1:48146)
0924 05:09:11.044743 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.044753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.044807 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.044989 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.044990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.044993 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:11.045384 (+ 391us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:11.045385 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:11.045391 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:11.045403 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:11.045640 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:11.045642 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:11.045642 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 05:09:11.045694 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:09:11.045706 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":88,"threads_started":1}
I20260924 05:09:11.050007 5464 tablet_bootstrap.cc:654] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:11.050101 5463 tablet_bootstrap.cc:654] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:11.050252 5252 tablet_service.cc:1467] Processing CreateTablet for tablet 5243908ebb2b498d949e271fce13024a (DEFAULT_TABLE table=test-table [id=151f70d3535d42b99e4182623da4e4a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:11.050360 5464 log.cc:824] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:11.050437 5463 log.cc:824] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:11.050571 5252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5243908ebb2b498d949e271fce13024a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:11.051090 5464 tablet_bootstrap.cc:492] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: No bootstrap required, opened a new log
I20260924 05:09:11.051162 5464 ts_tablet_manager.cc:1397] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:11.051297 5463 tablet_bootstrap.cc:492] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa: No bootstrap required, opened a new log
I20260924 05:09:11.051373 5463 ts_tablet_manager.cc:1397] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:11.053166 5463 raft_consensus.cc:348] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } }
I20260924 05:09:11.053305 5463 raft_consensus.cc:374] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:11.053349 5463 raft_consensus.cc:729] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d837a028f034a379375cb532cc783aa, State: Initialized, Role: FOLLOWER
I20260924 05:09:11.053453 5463 consensus_queue.cc:260] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } }
I20260924 05:09:11.053704 5463 ts_tablet_manager.cc:1428] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:09:11.053684 5464 raft_consensus.cc:348] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } }
I20260924 05:09:11.053845 5464 raft_consensus.cc:374] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:11.053876 5187 heartbeater.cc:503] Master 127.0.98.62:46411 was elected leader, sending a full tablet report...
I20260924 05:09:11.053889 5464 raft_consensus.cc:729] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d83e186d2e34cb6b51266dd5b3cac62, State: Initialized, Role: FOLLOWER
I20260924 05:09:11.053987 5464 consensus_queue.cc:260] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } }
I20260924 05:09:11.054214 5464 ts_tablet_manager.cc:1428] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:09:11.054230 5468 tablet_bootstrap.cc:492] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: Bootstrap starting.
I20260924 05:09:11.054316 5449 heartbeater.cc:503] Master 127.0.98.62:46411 was elected leader, sending a full tablet report...
I20260924 05:09:11.054946 5468 tablet_bootstrap.cc:654] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:11.055285 5468 log.cc:824] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:11.055912 5468 tablet_bootstrap.cc:492] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: No bootstrap required, opened a new log
I20260924 05:09:11.055961 5468 ts_tablet_manager.cc:1397] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:11.057816 5468 raft_consensus.cc:348] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } }
I20260924 05:09:11.057930 5468 raft_consensus.cc:374] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:11.057951 5468 raft_consensus.cc:729] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ca665fadcf423d9d5aa31adcfe8534, State: Initialized, Role: FOLLOWER
I20260924 05:09:11.058050 5468 consensus_queue.cc:260] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } }
I20260924 05:09:11.058250 5468 ts_tablet_manager.cc:1428] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:09:11.058319 5318 heartbeater.cc:503] Master 127.0.98.62:46411 was elected leader, sending a full tablet report...
I20260924 05:09:11.083593 5469 raft_consensus.cc:482] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:11.083741 5469 raft_consensus.cc:504] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } }
I20260924 05:09:11.084125 5469 leader_election.cc:288] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d837a028f034a379375cb532cc783aa (127.0.98.1:37649), e4ca665fadcf423d9d5aa31adcfe8534 (127.0.98.2:45435)
I20260924 05:09:11.085568 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.084301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37649 (local address 127.0.98.3:48527)
0924 05:09:11.084352 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.084363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.084400 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.084697 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.084698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.084701 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:11.085258 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:11.085258 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:11.085263 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:11.085272 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:11.085434 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:11.085434 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:11.085435 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:09:11.085478 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:09:11.085493 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:09:11.085577 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.084434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48527 (local address 127.0.98.1:37649)
0924 05:09:11.084499 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.084500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.084507 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:09:11.084544 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.084545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.084549 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.084645 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.085350 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.085351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.085351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.085355 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.085370 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.085375 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.085430 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.085488 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.085503 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:09:11.085577 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.084364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59243 (local address 127.0.98.2:45435)
0924 05:09:11.084418 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.084420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.084591 (+ 171us) server_negotiation.cc:407] Connection header received
0924 05:09:11.084619 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.084621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.084627 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.084702 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.085338 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.085340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.085340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.085345 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.085363 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.085367 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.085430 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.085497 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.085509 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:09:11.085587 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.084310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45435 (local address 127.0.98.3:59243)
0924 05:09:11.084521 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.084529 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.084573 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.084726 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.084726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.084728 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:11.085203 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:11.085203 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:11.085222 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:11.085238 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:11.085434 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:11.085434 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:11.085435 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:09:11.085484 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:09:11.085494 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":56,"thread_start_us":112,"threads_started":1}
I20260924 05:09:11.085914 5141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5243908ebb2b498d949e271fce13024a" candidate_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d837a028f034a379375cb532cc783aa" is_pre_election: true
I20260924 05:09:11.085942 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5243908ebb2b498d949e271fce13024a" candidate_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" is_pre_election: true
I20260924 05:09:11.086069 5272 raft_consensus.cc:2454] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d83e186d2e34cb6b51266dd5b3cac62 in term 0.
I20260924 05:09:11.086069 5141 raft_consensus.cc:2454] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d83e186d2e34cb6b51266dd5b3cac62 in term 0.
I20260924 05:09:11.086301 5336 leader_election.cc:302] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d83e186d2e34cb6b51266dd5b3cac62, e4ca665fadcf423d9d5aa31adcfe8534; no voters:
I20260924 05:09:11.086427 5469 raft_consensus.cc:2781] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:11.086479 5469 raft_consensus.cc:482] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:11.086511 5469 raft_consensus.cc:3037] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:11.087275 5469 raft_consensus.cc:504] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } }
I20260924 05:09:11.087435 5469 leader_election.cc:288] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [CANDIDATE]: Term 1 election: Requested vote from peers 2d837a028f034a379375cb532cc783aa (127.0.98.1:37649), e4ca665fadcf423d9d5aa31adcfe8534 (127.0.98.2:45435)
I20260924 05:09:11.087566 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5243908ebb2b498d949e271fce13024a" candidate_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ca665fadcf423d9d5aa31adcfe8534"
I20260924 05:09:11.087575 5141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5243908ebb2b498d949e271fce13024a" candidate_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d837a028f034a379375cb532cc783aa"
I20260924 05:09:11.087646 5272 raft_consensus.cc:3037] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:11.087670 5141 raft_consensus.cc:3037] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:11.088354 5141 raft_consensus.cc:2454] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d83e186d2e34cb6b51266dd5b3cac62 in term 1.
I20260924 05:09:11.088400 5272 raft_consensus.cc:2454] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d83e186d2e34cb6b51266dd5b3cac62 in term 1.
I20260924 05:09:11.088569 5338 leader_election.cc:302] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d837a028f034a379375cb532cc783aa, 2d83e186d2e34cb6b51266dd5b3cac62; no voters:
I20260924 05:09:11.088676 5469 raft_consensus.cc:2781] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:11.088832 5469 raft_consensus.cc:686] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 1 LEADER]: Becoming Leader. State: Replica: 2d83e186d2e34cb6b51266dd5b3cac62, State: Running, Role: LEADER
I20260924 05:09:11.088943 5469 consensus_queue.cc:237] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } }
I20260924 05:09:11.089704 4999 catalog_manager.cc:5495] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d83e186d2e34cb6b51266dd5b3cac62 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d83e186d2e34cb6b51266dd5b3cac62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 } health_report { overall_health: UNKNOWN } } }
I20260924 05:09:11.098796 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.092564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.0.98.3:44251)
0924 05:09:11.092604 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.092606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.092613 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:09:11.092677 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.092677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.092686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.092735 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.098529 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.098531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.098531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.098535 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.098549 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.098553 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.098603 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.098648 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.098667 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:09:11.100443 5272 raft_consensus.cc:1261] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [term 1 FOLLOWER]: Refusing update from remote peer 2d83e186d2e34cb6b51266dd5b3cac62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:11.100457 5141 raft_consensus.cc:1261] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [term 1 FOLLOWER]: Refusing update from remote peer 2d83e186d2e34cb6b51266dd5b3cac62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:11.100796 5469 consensus_queue.cc:1035] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ca665fadcf423d9d5aa31adcfe8534" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:11.100914 5473 consensus_queue.cc:1035] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d837a028f034a379375cb532cc783aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:11.103230 5478 mvcc.cc:205] Tried to move back new op lower bound from 7332767953304948736 to 7332767953260912640. Current Snapshot: MvccSnapshot[applied={T|T < 7332767953304948736}]
I20260924 05:09:11.103724 5477 mvcc.cc:205] Tried to move back new op lower bound from 7332767953304948736 to 7332767953260912640. Current Snapshot: MvccSnapshot[applied={T|T < 7332767953304948736}]
I20260924 05:09:11.105490 4998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:11.105574 4998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:11.106299 4998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff808a6d84ce474c9f53d90ed3751af5: Sending DeleteTablet for 3 replicas of tablet 5243908ebb2b498d949e271fce13024a
I20260924 05:09:11.106624 5252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5243908ebb2b498d949e271fce13024a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:11 UTC) from {username='slave'} at 127.0.0.1:48146
I20260924 05:09:11.106635 5121 tablet_service.cc:1514] Processing DeleteTablet for tablet 5243908ebb2b498d949e271fce13024a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:11 UTC) from {username='slave'} at 127.0.0.1:50316
I20260924 05:09:11.106860 5383 tablet_service.cc:1514] Processing DeleteTablet for tablet 5243908ebb2b498d949e271fce13024a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:11 UTC) from {username='slave'} at 127.0.0.1:56364
I20260924 05:09:11.106935 5486 tablet_replica.cc:330] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa: stopping tablet replica
I20260924 05:09:11.107019 5486 raft_consensus.cc:2229] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:11.107040 5488 tablet_replica.cc:330] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: stopping tablet replica
I20260924 05:09:11.107115 5486 raft_consensus.cc:2258] T 5243908ebb2b498d949e271fce13024a P 2d837a028f034a379375cb532cc783aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:11.107120 5488 raft_consensus.cc:2229] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:11.107208 5488 raft_consensus.cc:2258] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:11.107275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 5059
I20260924 05:09:11.108561 5488 ts_tablet_manager.cc:1905] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:11.110502 5488 ts_tablet_manager.cc:1918] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:11.110584 5488 log.cc:1196] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/5243908ebb2b498d949e271fce13024a
I20260924 05:09:11.110896 5487 tablet_replica.cc:330] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: stopping tablet replica
I20260924 05:09:11.110931 5488 ts_tablet_manager.cc:1939] T 5243908ebb2b498d949e271fce13024a P 2d83e186d2e34cb6b51266dd5b3cac62: Deleting consensus metadata
I20260924 05:09:11.111065 5487 raft_consensus.cc:2229] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:11.111207 5487 raft_consensus.cc:2258] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:11.111297 4986 catalog_manager.cc:4841] TS 2d83e186d2e34cb6b51266dd5b3cac62 (127.0.98.3:44251): tablet 5243908ebb2b498d949e271fce13024a (table test-table [id=151f70d3535d42b99e4182623da4e4a9]) successfully deleted
I20260924 05:09:11.111720 5487 ts_tablet_manager.cc:1905] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:09:11.112919 4985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37649 (error 108)
I20260924 05:09:11.112979 5487 ts_tablet_manager.cc:1918] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:11.113050 5487 log.cc:1196] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/5243908ebb2b498d949e271fce13024a
I20260924 05:09:11.113319 5487 ts_tablet_manager.cc:1939] T 5243908ebb2b498d949e271fce13024a P e4ca665fadcf423d9d5aa31adcfe8534: Deleting consensus metadata
I20260924 05:09:11.113633 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.113528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37649 (local address 127.0.0.1:50322)
0924 05:09:11.113571 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.113604 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 05:09:11.113682 4983 catalog_manager.cc:4841] TS e4ca665fadcf423d9d5aa31adcfe8534 (127.0.98.2:45435): tablet 5243908ebb2b498d949e271fce13024a (table test-table [id=151f70d3535d42b99e4182623da4e4a9]) successfully deleted
W20260924 05:09:11.113754 4985 catalog_manager.cc:4584] TS 2d837a028f034a379375cb532cc783aa (127.0.98.1:37649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5243908ebb2b498d949e271fce13024a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37649: connect: Connection refused (error 111)
I20260924 05:09:11.113917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 5190
I20260924 05:09:11.119568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 5321
I20260924 05:09:11.123766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 4968
2026-09-24T05:09:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (592 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 05:09:11.143074 392 mini_oidc.cc:152] Starting JWKS server
I20260924 05:09:11.144127 392 webserver.cc:460] Webserver started at https://127.0.0.1:43695/ using document root <none> and password file <none>
I20260924 05:09:11.144248 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 05:09:11.144423 392 webserver.cc:460] Webserver started at http://127.0.0.1:33051/ using document root <none> and password file <none>
2026-09-24T05:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:11Z Disabled control of system clock
I20260924 05:09:11.151870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34745
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34745
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43695/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/testchainca.cert with env {}
W20260924 05:09:11.230319 5495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:11.230521 5495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:11.230544 5495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:11.231673 5495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:11.231707 5495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:11.231720 5495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:11.231734 5495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:11.231745 5495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 05:09:11.231758 5495 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 05:09:11.231770 5495 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43695/jwks.json
I20260924 05:09:11.233222 5495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43695/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:11.233408 5495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:11.233654 5495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:11.238408 5500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:11.238399 5501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:11.238416 5495 server_base.cc:1034] running on GCE node
W20260924 05:09:11.238400 5503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:11.238921 5495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:11.239171 5495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:11.240336 5495 hybrid_clock.cc:648] HybridClock initialized: now 1790226551240318 us; error 31 us; skew 500 ppm
I20260924 05:09:11.241953 5495 webserver.cc:460] Webserver started at http://127.0.98.62:43531/ using document root <none> and password file <none>
I20260924 05:09:11.242195 5495 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:11.242276 5495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:11.242408 5495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:11.243467 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "ba5aac957dec4e2ea9a5b9da719a4c23"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "5340d5c1192a5a9b9ffb1b4a0ced08c6"
server_key_iv: "5ecd34e57191dd35e51aad8e62d9908f"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.243826 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba5aac957dec4e2ea9a5b9da719a4c23"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "5340d5c1192a5a9b9ffb1b4a0ced08c6"
server_key_iv: "5ecd34e57191dd35e51aad8e62d9908f"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.245081 5495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:11.245807 5509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:11.245986 5495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:11.246083 5495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "ba5aac957dec4e2ea9a5b9da719a4c23"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "5340d5c1192a5a9b9ffb1b4a0ced08c6"
server_key_iv: "5ecd34e57191dd35e51aad8e62d9908f"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.246167 5495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:11.263790 5495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:11.264106 5495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:11.264258 5495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:11.268501 5495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34745
I20260924 05:09:11.268539 5561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34745 every 8 connection(s)
I20260924 05:09:11.268903 5495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:11.269842 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:11.272046 5562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23: Bootstrap starting.
I20260924 05:09:11.272634 5562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:11.272915 5562 log.cc:824] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:11.273550 5562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23: No bootstrap required, opened a new log
I20260924 05:09:11.275149 5562 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5aac957dec4e2ea9a5b9da719a4c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260924 05:09:11.275278 5562 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:11.275300 5562 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5aac957dec4e2ea9a5b9da719a4c23, State: Initialized, Role: FOLLOWER
I20260924 05:09:11.275465 5562 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5aac957dec4e2ea9a5b9da719a4c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260924 05:09:11.275524 5562 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:11.275606 5562 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:11.275665 5562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:11.275800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 5495
I20260924 05:09:11.275913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:11.276090 392 external_mini_cluster.cc:1428] Setting key 796affeb330070b1b5d1316026c722ec
I20260924 05:09:11.276309 5562 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5aac957dec4e2ea9a5b9da719a4c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260924 05:09:11.276427 5562 leader_election.cc:302] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba5aac957dec4e2ea9a5b9da719a4c23; no voters:
I20260924 05:09:11.276610 5562 leader_election.cc:288] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:11.276695 5566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:11.276857 5566 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [term 1 LEADER]: Becoming Leader. State: Replica: ba5aac957dec4e2ea9a5b9da719a4c23, State: Running, Role: LEADER
I20260924 05:09:11.276878 5562 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:11.276944 5566 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5aac957dec4e2ea9a5b9da719a4c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260924 05:09:11.277424 5569 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5aac957dec4e2ea9a5b9da719a4c23. Latest consensus state: current_term: 1 leader_uuid: "ba5aac957dec4e2ea9a5b9da719a4c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5aac957dec4e2ea9a5b9da719a4c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } } }
I20260924 05:09:11.277508 5569 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:11.277652 5568 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba5aac957dec4e2ea9a5b9da719a4c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5aac957dec4e2ea9a5b9da719a4c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } } }
I20260924 05:09:11.277720 5568 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23 [sys.catalog]: This master's current role is: LEADER
W20260924 05:09:11.278653 5583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:09:11.278707 5583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:09:11.278887 5576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:11.279031 5576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:11.280515 5576 catalog_manager.cc:1347] Generated new cluster ID: c3e012ecc8fb4a9eb62ac1f3f02591f0
I20260924 05:09:11.280568 5576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:11.282856 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.276566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.0.98.62:34745)
0924 05:09:11.277363 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.277376 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.277416 (+ 40us) server_negotiation.cc:407] Connection header received
0924 05:09:11.277531 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.277539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.277622 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.278017 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.278115 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.279565 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.280689 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.281301 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.281423 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.282133 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.282136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.282137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.282351 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.282392 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.282407 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.282477 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.282542 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.282625 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":93,"threads_started":1}
I20260924 05:09:11.283748 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:09:11.291190 5576 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:11.291860 5576 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:11.299185 5576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba5aac957dec4e2ea9a5b9da719a4c23: Generated new TSK 0
I20260924 05:09:11.299351 5576 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 05:09:11.321867 5592 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 05:09:11.321969 5592 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 05:09:11.322432 5591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:11.307490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34745 (local address 127.0.0.1:33182)
0924 05:09:11.307688 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.307705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.307786 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.307979 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.307980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.307981 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 05:09:11.308046 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:11.308048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.308679 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.308680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:11.309304 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:11.309306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.309668 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.309672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:11.309734 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.309759 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 05:09:11.322134 (+ 12375us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 05:09:11.322203 (+ 69us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34745: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260924 05:09:11.322750 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.307560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.0.98.62:34745)
0924 05:09:11.307608 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.307610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.307764 (+ 154us) server_negotiation.cc:407] Connection header received
0924 05:09:11.307867 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.307868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.307880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.307974 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0924 05:09:11.308123 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.308599 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.309394 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.309586 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.309658 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.309881 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 05:09:11.321992 (+ 12111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 05:09:11.322655 (+ 663us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 05:09:11.322836 5567 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34745: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 05:09:11.323113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 5495
2026-09-24T05:09:11Z chronyd exiting
I20260924 05:09:11.358955 392 test_util.cc:175] -----------------------------------------------
I20260924 05:09:11.359035 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790226527740525-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (217 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 05:09:11.359783 392 mini_oidc.cc:152] Starting JWKS server
I20260924 05:09:11.360457 392 webserver.cc:460] Webserver started at https://127.0.0.1:36585/ using document root <none> and password file <none>
I20260924 05:09:11.360497 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 05:09:11.360675 392 webserver.cc:460] Webserver started at http://127.0.0.1:45319/ using document root <none> and password file <none>
2026-09-24T05:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:11Z Disabled control of system clock
I20260924 05:09:11.368095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37047
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37047
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36585/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/testchainca.cert with env {}
W20260924 05:09:11.450525 5598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:11.450748 5598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:11.450800 5598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:11.451960 5598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:11.452044 5598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:11.452068 5598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:11.452099 5598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:11.452123 5598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 05:09:11.452154 5598 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 05:09:11.452179 5598 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36585/jwks.json
I20260924 05:09:11.453637 5598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36585/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:11.453904 5598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:11.454115 5598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:11.459007 5604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:11.459023 5606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:11.459018 5603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:11.459173 5598 server_base.cc:1034] running on GCE node
I20260924 05:09:11.459398 5598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:11.459704 5598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:11.460896 5598 hybrid_clock.cc:648] HybridClock initialized: now 1790226551460881 us; error 44 us; skew 500 ppm
I20260924 05:09:11.462348 5598 webserver.cc:460] Webserver started at http://127.0.98.62:38921/ using document root <none> and password file <none>
I20260924 05:09:11.462595 5598 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:11.462667 5598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:11.462858 5598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:11.463912 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "569c11cab58e410596b9f716fafd840f"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "4f528a39f50e015cbdf17477b7dc6f5c"
server_key_iv: "89ac29a2e8ec923e0f16cf89d570b974"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.464279 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "569c11cab58e410596b9f716fafd840f"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "4f528a39f50e015cbdf17477b7dc6f5c"
server_key_iv: "89ac29a2e8ec923e0f16cf89d570b974"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.465571 5598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:11.466300 5612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:11.466471 5598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:11.466570 5598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "569c11cab58e410596b9f716fafd840f"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "4f528a39f50e015cbdf17477b7dc6f5c"
server_key_iv: "89ac29a2e8ec923e0f16cf89d570b974"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.466656 5598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:11.477847 5598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:11.478148 5598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:11.478296 5598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:11.482048 5598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37047
I20260924 05:09:11.482091 5664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37047 every 8 connection(s)
I20260924 05:09:11.482441 5598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:11.483383 5665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:11.485580 5665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f: Bootstrap starting.
I20260924 05:09:11.486150 5665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:11.486433 5665 log.cc:824] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:11.487134 5665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f: No bootstrap required, opened a new log
I20260924 05:09:11.488683 5665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569c11cab58e410596b9f716fafd840f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260924 05:09:11.488814 5665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:11.488835 5665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569c11cab58e410596b9f716fafd840f, State: Initialized, Role: FOLLOWER
I20260924 05:09:11.489022 5665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569c11cab58e410596b9f716fafd840f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260924 05:09:11.489082 5665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:11.489135 5665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:11.489187 5665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:11.489696 5665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569c11cab58e410596b9f716fafd840f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260924 05:09:11.489802 5665 leader_election.cc:302] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 569c11cab58e410596b9f716fafd840f; no voters:
I20260924 05:09:11.489977 5665 leader_election.cc:288] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:11.490010 5668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:11.490157 5668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [term 1 LEADER]: Becoming Leader. State: Replica: 569c11cab58e410596b9f716fafd840f, State: Running, Role: LEADER
I20260924 05:09:11.490224 5665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:11.490263 5668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569c11cab58e410596b9f716fafd840f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260924 05:09:11.490665 5670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569c11cab58e410596b9f716fafd840f. Latest consensus state: current_term: 1 leader_uuid: "569c11cab58e410596b9f716fafd840f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569c11cab58e410596b9f716fafd840f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } } }
I20260924 05:09:11.490793 5670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:11.490785 5669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "569c11cab58e410596b9f716fafd840f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569c11cab58e410596b9f716fafd840f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } } }
I20260924 05:09:11.491042 5669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:11.491046 5677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:11.491613 5677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:11.491901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 5598
I20260924 05:09:11.491998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:11.492187 392 external_mini_cluster.cc:1428] Setting key 6578a013df242b7697db5e5d9df64576
I20260924 05:09:11.493299 5677 catalog_manager.cc:1347] Generated new cluster ID: b7eec8f8c68e45fb8e4a3bc6782f8f68
I20260924 05:09:11.493368 5677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:11.500152 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.492758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.0.98.62:37047)
0924 05:09:11.494904 (+ 2146us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.494912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.494943 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:09:11.495057 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.495064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.495129 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.495260 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.495498 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.496851 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.497991 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.498579 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.498687 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.499472 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.499476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.499476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.499673 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.499713 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.499726 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.499794 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.499886 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.499957 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2078,"thread_start_us":72,"threads_started":1}
I20260924 05:09:11.501091 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:09:11.508695 5677 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:11.509188 5677 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:11.517416 5677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 569c11cab58e410596b9f716fafd840f: Generated new TSK 0
I20260924 05:09:11.517597 5677 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 05:09:11.543608 5695 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 05:09:11.543697 5695 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 05:09:11.544001 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.522051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.0.98.62:37047)
0924 05:09:11.522110 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.522112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.522238 (+ 126us) server_negotiation.cc:407] Connection header received
0924 05:09:11.522319 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.522320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.522332 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.522417 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0924 05:09:11.522538 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.523092 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.523877 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.524078 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.524152 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.524336 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 05:09:11.543746 (+ 19410us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 05:09:11.543908 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57950: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 05:09:11.544057 5687 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57950: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 05:09:11.544059 5694 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:11.521985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37047 (local address 127.0.0.1:57950)
0924 05:09:11.522170 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.522179 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.522256 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.522416 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.522417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.522418 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 05:09:11.522479 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:11.522481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.523169 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.523171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:11.523767 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:11.523769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.524149 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.524152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:11.524216 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.524222 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 05:09:11.543856 (+ 19634us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 05:09:11.543965 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260924 05:09:11.544410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 5598
2026-09-24T05:09:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (221 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 05:09:11.581605 392 mini_oidc.cc:152] Starting JWKS server
I20260924 05:09:11.582278 392 webserver.cc:460] Webserver started at https://127.0.0.1:33349/ using document root <none> and password file <none>
I20260924 05:09:11.582329 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 05:09:11.582522 392 webserver.cc:460] Webserver started at http://127.0.0.1:39985/ using document root <none> and password file <none>
2026-09-24T05:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:11Z Disabled control of system clock
I20260924 05:09:11.589792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36067
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36067
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33349/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 05:09:11.669003 5701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:11.669242 5701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:11.669281 5701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:11.670459 5701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:11.670523 5701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:11.670547 5701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:11.670579 5701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:11.670603 5701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 05:09:11.670634 5701 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 05:09:11.670662 5701 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33349/jwks.json
I20260924 05:09:11.672075 5701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33349/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:11.672356 5701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:11.672640 5701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:09:11.677397 5701 server_base.cc:1034] running on GCE node
W20260924 05:09:11.677366 5707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:11.677350 5706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:11.677341 5709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:11.677742 5701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:11.678059 5701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:11.679217 5701 hybrid_clock.cc:648] HybridClock initialized: now 1790226551679206 us; error 46 us; skew 500 ppm
I20260924 05:09:11.680433 5701 webserver.cc:460] Webserver started at http://127.0.98.62:39737/ using document root <none> and password file <none>
I20260924 05:09:11.680708 5701 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:11.680778 5701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:11.680934 5701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:11.681998 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "51b40ad317774bc1a00b334211b74a15"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "8613b83752fe9182dab199062baf77a7"
server_key_iv: "2cd5b2229b4836f0c32124ae3bdc345e"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.682389 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "51b40ad317774bc1a00b334211b74a15"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "8613b83752fe9182dab199062baf77a7"
server_key_iv: "2cd5b2229b4836f0c32124ae3bdc345e"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.683801 5701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:11.684490 5715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:11.684657 5701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:11.684753 5701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "51b40ad317774bc1a00b334211b74a15"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "8613b83752fe9182dab199062baf77a7"
server_key_iv: "2cd5b2229b4836f0c32124ae3bdc345e"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.684836 5701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:11.710902 5701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:11.711237 5701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:11.711402 5701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:11.715330 5701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36067
I20260924 05:09:11.715369 5767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36067 every 8 connection(s)
I20260924 05:09:11.715737 5701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:11.716380 5768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:11.719002 5768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15: Bootstrap starting.
I20260924 05:09:11.719561 5768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:11.719854 5768 log.cc:824] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:11.720481 5768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15: No bootstrap required, opened a new log
I20260924 05:09:11.722025 5768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b40ad317774bc1a00b334211b74a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } }
I20260924 05:09:11.722164 5768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:11.722224 5768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b40ad317774bc1a00b334211b74a15, State: Initialized, Role: FOLLOWER
I20260924 05:09:11.722407 5768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b40ad317774bc1a00b334211b74a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } }
I20260924 05:09:11.722493 5768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:11.722545 5768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:11.722596 5768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:11.723121 5768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b40ad317774bc1a00b334211b74a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } }
I20260924 05:09:11.723232 5768 leader_election.cc:302] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51b40ad317774bc1a00b334211b74a15; no voters:
I20260924 05:09:11.723400 5768 leader_election.cc:288] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:11.723460 5771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:11.723624 5771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [term 1 LEADER]: Becoming Leader. State: Replica: 51b40ad317774bc1a00b334211b74a15, State: Running, Role: LEADER
I20260924 05:09:11.723680 5768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:11.723722 5771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b40ad317774bc1a00b334211b74a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } }
I20260924 05:09:11.723877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 5701
I20260924 05:09:11.723980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:11.724179 392 external_mini_cluster.cc:1428] Setting key ac39921d78d4bba8f09bb32c01855d8d
I20260924 05:09:11.724242 5773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51b40ad317774bc1a00b334211b74a15. Latest consensus state: current_term: 1 leader_uuid: "51b40ad317774bc1a00b334211b74a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b40ad317774bc1a00b334211b74a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } } }
I20260924 05:09:11.724330 5773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:11.724565 5780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:11.724983 5772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51b40ad317774bc1a00b334211b74a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b40ad317774bc1a00b334211b74a15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } } }
I20260924 05:09:11.725077 5780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:11.725121 5772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:11.727120 5780 catalog_manager.cc:1347] Generated new cluster ID: a5943a13148849958fe0a2e3f2d19466
I20260924 05:09:11.727173 5780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:11.730130 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.724646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.0.98.62:36067)
0924 05:09:11.725446 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.725454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.725484 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:09:11.725753 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.725762 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.725859 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.726010 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.726228 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.727439 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.728224 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.728647 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.728777 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.729461 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.729465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.729465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.729656 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.729687 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.729698 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.729777 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.729855 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.729939 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":67,"threads_started":1}
I20260924 05:09:11.731026 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:09:11.746049 5780 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:11.746553 5780 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:11.750905 5780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51b40ad317774bc1a00b334211b74a15: Generated new TSK 0
I20260924 05:09:11.751091 5780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:09:11.763880 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.760347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.0.98.62:36067)
0924 05:09:11.760392 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.760394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.760486 (+ 92us) server_negotiation.cc:407] Connection header received
0924 05:09:11.760526 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.760527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.760539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.760623 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0924 05:09:11.760724 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.761486 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.762222 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.762525 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.762596 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.762741 (+ 145us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 05:09:11.763680 (+ 939us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 05:09:11.763812 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 05:09:11.763937 5784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 05:09:11.765793 5797 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:11.760253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36067 (local address 127.0.0.1:34682)
0924 05:09:11.760406 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.760416 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.760481 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.760620 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.760621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.760622 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 05:09:11.760664 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:11.760666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.761556 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.761556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:11.762134 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:11.762135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.762599 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.762602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:11.762638 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.762642 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 05:09:11.763782 (+ 1140us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 05:09:11.765692 (+ 1910us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36067: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":1734,"thread_start_us":106,"threads_started":1}
I20260924 05:09:11.766132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 5701
E20260924 05:09:11.775241 5798 webserver.cc:533] Webserver: sslize failed:
E20260924 05:09:11.775297 5798 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T05:09:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T05:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:11Z Disabled control of system clock
I20260924 05:09:11.811496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38057
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38057
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/testchainca.cert with env {}
W20260924 05:09:11.889606 5802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:11.889801 5802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:11.889822 5802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:11.891026 5802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:11.891067 5802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:11.891079 5802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:11.891093 5802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:11.891135 5802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 05:09:11.891166 5802 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 05:09:11.891182 5802 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 05:09:11.892657 5802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:11.892906 5802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:11.893122 5802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:11.898048 5810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:11.898106 5802 server_base.cc:1034] running on GCE node
W20260924 05:09:11.898156 5807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:11.898109 5808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:11.898458 5802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:11.898743 5802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:11.899935 5802 hybrid_clock.cc:648] HybridClock initialized: now 1790226551899923 us; error 29 us; skew 500 ppm
I20260924 05:09:11.901177 5802 webserver.cc:460] Webserver started at http://127.0.98.62:36045/ using document root <none> and password file <none>
I20260924 05:09:11.901428 5802 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:11.901499 5802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:11.901641 5802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:11.902647 5802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "5895dbf8a06a4b098a977a5c9b459c28"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "45cef65bf1651200987de235a9896777"
server_key_iv: "c6d20678a8e3ea8fe5f422a44edd452a"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.903053 5802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "5895dbf8a06a4b098a977a5c9b459c28"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "45cef65bf1651200987de235a9896777"
server_key_iv: "c6d20678a8e3ea8fe5f422a44edd452a"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.904284 5802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:11.904965 5816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:11.905138 5802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:11.905231 5802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "5895dbf8a06a4b098a977a5c9b459c28"
format_stamp: "Formatted at 2026-09-24 05:09:11 on dist-test-slave-mr2f"
server_key: "45cef65bf1651200987de235a9896777"
server_key_iv: "c6d20678a8e3ea8fe5f422a44edd452a"
server_key_version: "encryptionkey@0"
I20260924 05:09:11.905300 5802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:11.923265 5802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:11.923573 5802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:11.923728 5802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:11.927520 5802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38057
I20260924 05:09:11.927574 5868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38057 every 8 connection(s)
I20260924 05:09:11.927904 5802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:11.928802 5869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:11.931052 5869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28: Bootstrap starting.
I20260924 05:09:11.931610 5869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:11.931900 5869 log.cc:824] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:11.932543 5869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28: No bootstrap required, opened a new log
I20260924 05:09:11.934078 5869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5895dbf8a06a4b098a977a5c9b459c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38057 } }
I20260924 05:09:11.934216 5869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:11.934264 5869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5895dbf8a06a4b098a977a5c9b459c28, State: Initialized, Role: FOLLOWER
I20260924 05:09:11.934419 5869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5895dbf8a06a4b098a977a5c9b459c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38057 } }
I20260924 05:09:11.934481 5869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:11.934521 5869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:11.934572 5869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:11.935089 5869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5895dbf8a06a4b098a977a5c9b459c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38057 } }
I20260924 05:09:11.935215 5869 leader_election.cc:302] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5895dbf8a06a4b098a977a5c9b459c28; no voters:
I20260924 05:09:11.935386 5869 leader_election.cc:288] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:11.935405 5872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:11.935598 5872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [term 1 LEADER]: Becoming Leader. State: Replica: 5895dbf8a06a4b098a977a5c9b459c28, State: Running, Role: LEADER
I20260924 05:09:11.935648 5869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:11.935647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 5802
I20260924 05:09:11.935748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:11.935708 5872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5895dbf8a06a4b098a977a5c9b459c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38057 } }
I20260924 05:09:11.935993 392 external_mini_cluster.cc:1428] Setting key 6fe4dc71db4f382ab257c81f83a34d5d
I20260924 05:09:11.936092 5872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5895dbf8a06a4b098a977a5c9b459c28. Latest consensus state: current_term: 1 leader_uuid: "5895dbf8a06a4b098a977a5c9b459c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5895dbf8a06a4b098a977a5c9b459c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38057 } } }
I20260924 05:09:11.936177 5872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:11.936115 5873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5895dbf8a06a4b098a977a5c9b459c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5895dbf8a06a4b098a977a5c9b459c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38057 } } }
I20260924 05:09:11.936335 5873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:11.936775 5880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:11.937250 5880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:11.938943 5880 catalog_manager.cc:1347] Generated new cluster ID: 5495b62318a5410e8c0dc8faa73b553f
I20260924 05:09:11.938995 5880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:11.941962 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.936567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.0.98.62:38057)
0924 05:09:11.936771 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.936780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.936810 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:09:11.936965 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.936972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.937047 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.937189 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:11.937441 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.938704 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.939806 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.940392 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.940512 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.941196 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:11.941200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:11.941201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:11.941395 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:11.941435 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:11.941448 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:11.941510 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:11.941607 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 05:09:11.941759 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260924 05:09:11.942781 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:09:11.951407 5880 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:11.952105 5880 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:11.961045 5880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5895dbf8a06a4b098a977a5c9b459c28: Generated new TSK 0
I20260924 05:09:11.961231 5880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:09:12.461472 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:11.968262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.0.98.62:38057)
0924 05:09:11.968323 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:11.968325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:11.968435 (+ 110us) server_negotiation.cc:407] Connection header received
0924 05:09:11.968506 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:11.968506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:11.968517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:11.968570 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0924 05:09:11.968712 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.969203 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.970325 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.970516 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.970565 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.970755 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 05:09:12.461233 (+490478us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 05:09:12.461411 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58890: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 05:09:12.461562 5883 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58890: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 05:09:12.461697 5898 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:11.968204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38057 (local address 127.0.0.1:58890)
0924 05:09:11.968362 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:11.968373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:11.968438 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:11.968581 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:11.968582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:11.968584 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 05:09:11.968655 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:11.968657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.969283 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.969284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:11.970243 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:11.970246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:11.970589 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:11.970593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:11.970659 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:11.970665 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 05:09:12.461386 (+490721us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 05:09:12.461588 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38057: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260924 05:09:12.462021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 5802
2026-09-24T05:09:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (674 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (84 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:12 dist-test-slave-mr2f krb5kdc[5910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:12 dist-test-slave-mr2f krb5kdc[5910](info): set up 2 sockets
Sep 24 05:09:12 dist-test-slave-mr2f krb5kdc[5910](info): commencing operation
krb5kdc: starting...
W20260924 05:09:13.662914 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:13 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:09:13.777849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35987
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:13.855788 5923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:13.856022 5923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:13.856061 5923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:13.857187 5923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:13.857254 5923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:13.857278 5923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:13.857307 5923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:13.857339 5923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:13.858861 5923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:13.859131 5923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:13.859355 5923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:13.863951 5931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:13.863965 5929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:13.864166 5923 server_base.cc:1034] running on GCE node
W20260924 05:09:13.863965 5928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:13.864477 5923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:13.864744 5923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:13.865895 5923 hybrid_clock.cc:648] HybridClock initialized: now 1790226553865873 us; error 40 us; skew 500 ppm
Sep 24 05:09:13 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:13.868301 5923 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:09:13.868670 5923 webserver.cc:460] Webserver started at http://127.0.98.62:44323/ using document root <none> and password file <none>
I20260924 05:09:13.868847 5923 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:13.868886 5923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:13.868973 5923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:13.869987 5923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "094c81d2e23b42c383f07005420efdb1"
format_stamp: "Formatted at 2026-09-24 05:09:13 on dist-test-slave-mr2f"
server_key: "aa09733570a71ca6bd0fee4364c35311"
server_key_iv: "345a33b33bf229fc1d6acbe46d37b2ef"
server_key_version: "encryptionkey@0"
I20260924 05:09:13.870309 5923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "094c81d2e23b42c383f07005420efdb1"
format_stamp: "Formatted at 2026-09-24 05:09:13 on dist-test-slave-mr2f"
server_key: "aa09733570a71ca6bd0fee4364c35311"
server_key_iv: "345a33b33bf229fc1d6acbe46d37b2ef"
server_key_version: "encryptionkey@0"
I20260924 05:09:13.871663 5923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:13.872354 5938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:13.872519 5923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:13.872617 5923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "094c81d2e23b42c383f07005420efdb1"
format_stamp: "Formatted at 2026-09-24 05:09:13 on dist-test-slave-mr2f"
server_key: "aa09733570a71ca6bd0fee4364c35311"
server_key_iv: "345a33b33bf229fc1d6acbe46d37b2ef"
server_key_version: "encryptionkey@0"
I20260924 05:09:13.872699 5923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:13.886909 5923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:13.888604 5923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:13.888759 5923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:13.892632 5923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35987
I20260924 05:09:13.892666 5990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35987 every 8 connection(s)
I20260924 05:09:13.893170 5923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:13.893810 5991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:13.896041 5991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1: Bootstrap starting.
I20260924 05:09:13.896596 5991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:13.896880 5991 log.cc:824] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:13.897496 5991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1: No bootstrap required, opened a new log
I20260924 05:09:13.899055 5991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094c81d2e23b42c383f07005420efdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35987 } }
I20260924 05:09:13.899190 5991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:13.899253 5991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094c81d2e23b42c383f07005420efdb1, State: Initialized, Role: FOLLOWER
I20260924 05:09:13.899436 5991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094c81d2e23b42c383f07005420efdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35987 } }
I20260924 05:09:13.899526 5991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:13.899580 5991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:13.899631 5991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:13.900482 5991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094c81d2e23b42c383f07005420efdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35987 } }
I20260924 05:09:13.900564 5991 leader_election.cc:302] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 094c81d2e23b42c383f07005420efdb1; no voters:
I20260924 05:09:13.900709 5991 leader_election.cc:288] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:13.900799 5994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:13.900954 5994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [term 1 LEADER]: Becoming Leader. State: Replica: 094c81d2e23b42c383f07005420efdb1, State: Running, Role: LEADER
I20260924 05:09:13.900969 5991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:13.901051 5994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094c81d2e23b42c383f07005420efdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35987 } }
I20260924 05:09:13.901408 5995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "094c81d2e23b42c383f07005420efdb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094c81d2e23b42c383f07005420efdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35987 } } }
I20260924 05:09:13.901420 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094c81d2e23b42c383f07005420efdb1. Latest consensus state: current_term: 1 leader_uuid: "094c81d2e23b42c383f07005420efdb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094c81d2e23b42c383f07005420efdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35987 } } }
I20260924 05:09:13.901510 5995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:13.901515 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:13.901713 6003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:13.902143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 5923
I20260924 05:09:13.902256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:13.902419 392 external_mini_cluster.cc:1428] Setting key 8023591f5a8d368c9725c4694ee9793b
I20260924 05:09:13.902865 6003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:13.904315 6003 catalog_manager.cc:1347] Generated new cluster ID: 3477f09682d34b0e95fcd524c0304726
I20260924 05:09:13.904371 6003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:09:13 dist-test-slave-mr2f krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:13.911738 6003 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:13.912398 6003 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:13.913923 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:13.903030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50916 (local address 127.0.98.62:35987)
0924 05:09:13.903229 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:13.903237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:13.903447 (+ 210us) server_negotiation.cc:407] Connection header received
0924 05:09:13.903553 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:13.903560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:13.903610 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:13.903751 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:13.904486 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:13.905856 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:13.906873 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:13.907513 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:13.907623 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:13.910320 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:13.910324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:13.910324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:13.910337 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:13.912357 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:13.912754 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:13.912757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:13.912758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:13.912854 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:13.913070 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:13.913071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:13.913071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:13.913302 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:13.913457 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:13.913565 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 05:09:13.913715 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 05:09:13.914885 6003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 094c81d2e23b42c383f07005420efdb1: Generated new TSK 0
I20260924 05:09:13.915031 6003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:09:13.943625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35987
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:14.021544 6017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:14.021766 6017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:14.021804 6017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:14.022974 6017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:14.023043 6017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:14.023089 6017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:14.024554 6017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:14.024816 6017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:14.025053 6017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:14.030056 6023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:14.030040 6025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:14.030028 6022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:14.030227 6017 server_base.cc:1034] running on GCE node
I20260924 05:09:14.030457 6017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:14.030714 6017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:14.031898 6017 hybrid_clock.cc:648] HybridClock initialized: now 1790226554031888 us; error 32 us; skew 500 ppm
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.034224 6017 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:09:14.034677 6017 webserver.cc:460] Webserver started at http://127.0.98.1:44509/ using document root <none> and password file <none>
I20260924 05:09:14.034945 6017 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:14.035017 6017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:14.035164 6017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:14.036201 6017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "9f06b5cb50ec455caadea22fffd5e944"
format_stamp: "Formatted at 2026-09-24 05:09:14 on dist-test-slave-mr2f"
server_key: "1d18680e9104dbb61d93cba6c8cb50e1"
server_key_iv: "39cc8d9c4e44b7bc35063b2a30f4eec7"
server_key_version: "encryptionkey@0"
I20260924 05:09:14.036544 6017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9f06b5cb50ec455caadea22fffd5e944"
format_stamp: "Formatted at 2026-09-24 05:09:14 on dist-test-slave-mr2f"
server_key: "1d18680e9104dbb61d93cba6c8cb50e1"
server_key_iv: "39cc8d9c4e44b7bc35063b2a30f4eec7"
server_key_version: "encryptionkey@0"
I20260924 05:09:14.037873 6017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:14.038563 6032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:14.038755 6017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:14.038872 6017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "9f06b5cb50ec455caadea22fffd5e944"
format_stamp: "Formatted at 2026-09-24 05:09:14 on dist-test-slave-mr2f"
server_key: "1d18680e9104dbb61d93cba6c8cb50e1"
server_key_iv: "39cc8d9c4e44b7bc35063b2a30f4eec7"
server_key_version: "encryptionkey@0"
I20260924 05:09:14.038928 6017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:14.065958 6017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:14.068392 6017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:14.068504 6017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:14.068688 6017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:14.068950 6017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:14.068979 6017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:14.069001 6017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:14.069015 6017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:14.074707 6017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43291
I20260924 05:09:14.074770 6145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43291 every 8 connection(s)
I20260924 05:09:14.075125 6017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:09:14.078082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 6017
I20260924 05:09:14.078205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:14.078387 392 external_mini_cluster.cc:1428] Setting key 37324224bb2ef19c37b9e18ce2e17acb
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:14.086050 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:14.075802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54879 (local address 127.0.98.62:35987)
0924 05:09:14.075843 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:14.075845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:14.076064 (+ 219us) server_negotiation.cc:407] Connection header received
0924 05:09:14.077087 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:14.077089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:14.077103 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:14.077153 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:14.077762 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.078355 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.079655 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.079880 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.079955 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:14.082512 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:14.082513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:14.082514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:14.082520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:14.084519 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.085070 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.085071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.085072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.085134 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.085390 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.085391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.085391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.085507 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:14.085580 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:14.085861 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 05:09:14.085958 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:09:14.086236 6148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:14.075773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35987 (local address 127.0.98.1:54879)
0924 05:09:14.075909 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:14.075973 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:14.076968 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:14.077220 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:14.077224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:14.077472 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:14.077659 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:14.077663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.078446 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.078447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:14.079552 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:14.079555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.079974 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.079975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:14.080125 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:14.080620 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:14.080629 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:14.082379 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:14.084612 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:14.084636 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:14.084638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:14.084982 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:14.085201 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:14.085202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:14.085202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:14.085334 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:14.085585 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:14.085591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:14.085726 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 05:09:14.085827 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 05:09:14.085978 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260924 05:09:14.086705 6146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35987
I20260924 05:09:14.086824 6146 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:09:14.087023 6146 heartbeater.cc:511] Master 127.0.98.62:35987 requested a full tablet report, sending...
I20260924 05:09:14.087555 5955 ts_manager.cc:194] Registered new tserver with Master: 9f06b5cb50ec455caadea22fffd5e944 (127.0.98.1:43291)
I20260924 05:09:14.088186 5955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:09:14.103209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35987
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:14.181257 6151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:14.181459 6151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:14.181483 6151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:14.182611 6151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:14.182652 6151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:14.182678 6151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:14.184160 6151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:14.184408 6151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:14.184640 6151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:14.189508 6156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:14.189527 6159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:14.189589 6151 server_base.cc:1034] running on GCE node
W20260924 05:09:14.189505 6157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:14.189905 6151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:14.190153 6151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:14.191318 6151 hybrid_clock.cc:648] HybridClock initialized: now 1790226554191312 us; error 37 us; skew 500 ppm
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.193670 6151 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:09:14.194077 6151 webserver.cc:460] Webserver started at http://127.0.98.2:34927/ using document root <none> and password file <none>
I20260924 05:09:14.194301 6151 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:14.194368 6151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:14.194511 6151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:14.195581 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "b4d2cb4f49b54eaf80dadeded41959ed"
format_stamp: "Formatted at 2026-09-24 05:09:14 on dist-test-slave-mr2f"
server_key: "d8ed9b2be253a8e2a09ffa43c44cdc8c"
server_key_iv: "12151b380b14680e4065fe93f270a1b3"
server_key_version: "encryptionkey@0"
I20260924 05:09:14.195956 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b4d2cb4f49b54eaf80dadeded41959ed"
format_stamp: "Formatted at 2026-09-24 05:09:14 on dist-test-slave-mr2f"
server_key: "d8ed9b2be253a8e2a09ffa43c44cdc8c"
server_key_iv: "12151b380b14680e4065fe93f270a1b3"
server_key_version: "encryptionkey@0"
I20260924 05:09:14.197225 6151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:09:14.197921 6166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:14.198091 6151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:14.198199 6151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "b4d2cb4f49b54eaf80dadeded41959ed"
format_stamp: "Formatted at 2026-09-24 05:09:14 on dist-test-slave-mr2f"
server_key: "d8ed9b2be253a8e2a09ffa43c44cdc8c"
server_key_iv: "12151b380b14680e4065fe93f270a1b3"
server_key_version: "encryptionkey@0"
I20260924 05:09:14.198282 6151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:14.214602 6151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:14.216706 6151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:14.216861 6151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:14.217087 6151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:14.217424 6151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:14.217497 6151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:14.217540 6151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:14.217567 6151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:14.223080 6151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43063
I20260924 05:09:14.223121 6279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43063 every 8 connection(s)
I20260924 05:09:14.223430 6151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:14.227677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 6151
I20260924 05:09:14.227807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:14.227983 392 external_mini_cluster.cc:1428] Setting key f2c7b101c87982c88ab5d069ee66f6a6
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:14.232951 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:14.224137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46871 (local address 127.0.98.62:35987)
0924 05:09:14.224233 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:14.224235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:14.224394 (+ 159us) server_negotiation.cc:407] Connection header received
0924 05:09:14.225352 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:14.225354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:14.225363 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:14.225411 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:14.225966 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.226470 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.227463 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.227774 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.227850 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:14.229973 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:14.229974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:14.229975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:14.229982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:14.231371 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.231867 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.231869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.231869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.231936 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.232222 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.232222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.232223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.232318 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:14.232368 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:14.232722 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 05:09:14.232808 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 05:09:14.232960 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:14.224093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35987 (local address 127.0.98.2:46871)
0924 05:09:14.224271 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:14.224306 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:14.225250 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:14.225472 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:14.225475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:14.225715 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:14.225897 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:14.225901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.226548 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.226549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:14.227393 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:14.227395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.227893 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.227896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:14.227999 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:14.228504 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:14.228513 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:14.229876 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:14.231453 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:14.231459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:14.231461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:14.231781 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:14.232005 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:14.232006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:14.232007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:14.232153 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:14.232386 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:14.232392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:14.232551 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 05:09:14.232688 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 05:09:14.232803 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260924 05:09:14.233453 6280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35987
I20260924 05:09:14.233546 6280 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:14.233714 6280 heartbeater.cc:511] Master 127.0.98.62:35987 requested a full tablet report, sending...
I20260924 05:09:14.234126 5955 ts_manager.cc:194] Registered new tserver with Master: b4d2cb4f49b54eaf80dadeded41959ed (127.0.98.2:43063)
I20260924 05:09:14.234694 5955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46871
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:09:14.255427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35987
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:14.334479 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:14.334666 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:14.334688 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:14.335841 6285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:14.335883 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:14.335909 6285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:14.337262 6285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:14.337436 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:14.337590 6285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:14.342264 6291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:14.342269 6290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:14.342279 6293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:14.342561 6285 server_base.cc:1034] running on GCE node
I20260924 05:09:14.342825 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:14.343065 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:14.344249 6285 hybrid_clock.cc:648] HybridClock initialized: now 1790226554344237 us; error 42 us; skew 500 ppm
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.346592 6285 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:09:14.347059 6285 webserver.cc:460] Webserver started at http://127.0.98.3:46703/ using document root <none> and password file <none>
I20260924 05:09:14.347297 6285 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:14.347365 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:14.347498 6285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:14.348515 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6d1771c3826466aa1151e34ab38dfe5"
format_stamp: "Formatted at 2026-09-24 05:09:14 on dist-test-slave-mr2f"
server_key: "e74f7c60400501ee20282ff7a51d2187"
server_key_iv: "edae5036c6c76b17dad10b6981e33498"
server_key_version: "encryptionkey@0"
I20260924 05:09:14.348866 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6d1771c3826466aa1151e34ab38dfe5"
format_stamp: "Formatted at 2026-09-24 05:09:14 on dist-test-slave-mr2f"
server_key: "e74f7c60400501ee20282ff7a51d2187"
server_key_iv: "edae5036c6c76b17dad10b6981e33498"
server_key_version: "encryptionkey@0"
I20260924 05:09:14.350159 6285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:14.350847 6300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:14.351015 6285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:14.351107 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "b6d1771c3826466aa1151e34ab38dfe5"
format_stamp: "Formatted at 2026-09-24 05:09:14 on dist-test-slave-mr2f"
server_key: "e74f7c60400501ee20282ff7a51d2187"
server_key_iv: "edae5036c6c76b17dad10b6981e33498"
server_key_version: "encryptionkey@0"
I20260924 05:09:14.351202 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:14.378238 6285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:14.380182 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:14.380338 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:14.380568 6285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:14.380869 6285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:14.380923 6285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:14.380962 6285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:14.380990 6285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:14.386448 6285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38573
I20260924 05:09:14.386547 6413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38573 every 8 connection(s)
I20260924 05:09:14.386906 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:14.389973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 6285
I20260924 05:09:14.390074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:14.390249 392 external_mini_cluster.cc:1428] Setting key cd65564a6a2f2bc40a0205dd8f370bad
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:14.396536 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:14.387670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37133 (local address 127.0.98.62:35987)
0924 05:09:14.387748 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:14.387751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:14.387934 (+ 183us) server_negotiation.cc:407] Connection header received
0924 05:09:14.388854 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:14.388855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:14.388866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:14.388923 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:14.389559 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.390346 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.391674 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.391854 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.391911 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:14.393896 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:14.393897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:14.393897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:14.393904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:14.395319 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.395754 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.395755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.395755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.395814 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.396031 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.396031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.396032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.396132 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:14.396198 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:14.396411 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 05:09:14.396472 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 05:09:14.396670 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:14.387626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35987 (local address 127.0.98.3:37133)
0924 05:09:14.387805 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:14.387843 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:14.388746 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:14.388992 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:14.388996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:14.389235 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:14.389444 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:14.389448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.390455 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.390456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:14.391601 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:14.391604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.391932 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.391933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:14.392031 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:14.392468 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:14.392476 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:14.393796 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:14.395384 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:14.395390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:14.395392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:14.395596 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:14.395882 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:14.395882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:14.395883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:14.395973 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:14.396190 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:14.396194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:14.396300 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 05:09:14.396382 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 05:09:14.396492 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260924 05:09:14.397166 6414 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35987
I20260924 05:09:14.397243 6414 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:14.397459 6414 heartbeater.cc:511] Master 127.0.98.62:35987 requested a full tablet report, sending...
I20260924 05:09:14.397806 5955 ts_manager.cc:194] Registered new tserver with Master: b6d1771c3826466aa1151e34ab38dfe5 (127.0.98.3:38573)
I20260924 05:09:14.398128 5955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37133
I20260924 05:09:14.402182 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/test-admin.keytab.
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.416865 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 05:09:14.418344 6421 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:14.417626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.0.98.1:43291)
0924 05:09:14.417781 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:14.417786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:14.417795 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:14.417821 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:14.417826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:14.417844 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:14.417927 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:14.418263 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49440: BlockingRecv error: recv got EOF from 127.0.0.1:49440 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
W20260924 05:09:14.418421 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5cc)
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.419363 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.420188 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:09:14.427520 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:14.420682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.0.98.1:43291)
0924 05:09:14.420756 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:14.420758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:14.420838 (+ 80us) server_negotiation.cc:407] Connection header received
0924 05:09:14.420910 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:14.420911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:14.420921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:14.420975 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:14.421329 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.422099 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.422880 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.423193 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.423254 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:14.425104 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:14.425111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:14.425111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:14.425117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:14.426480 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.426898 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.426900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.426901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.426953 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.427123 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.427123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.427124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.427237 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:14.427329 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:14.427397 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 05:09:14.427454 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 05:09:14.429163 6421 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:14.428705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.0.98.1:43291)
0924 05:09:14.428738 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:14.428739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:14.428777 (+ 38us) server_negotiation.cc:407] Connection header received
0924 05:09:14.428839 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:14.428840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:14.428848 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:14.428897 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:14.429109 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49460: BlockingRecv error: recv got EOF from 127.0.0.1:49460 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:09:14.429239 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.430105 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.431000 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:09:14.437600 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:14.431425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.0.98.1:43291)
0924 05:09:14.431459 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:14.431461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:14.431470 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:14.431541 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:14.431542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:14.431552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:14.431621 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:14.431997 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.432457 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.433112 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.433301 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.433389 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:14.435360 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:14.435361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:14.435362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:14.435368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:14.436540 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.436890 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.436891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.436891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.436947 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.437144 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.437145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.437145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.437239 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:14.437333 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:14.437444 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 05:09:14.437515 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 05:09:14.439107 6421 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:14.438586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.0.98.1:43291)
0924 05:09:14.438617 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:14.438618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:14.438651 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:09:14.438717 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:14.438717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:14.438723 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:14.438821 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:14.439069 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49480: BlockingRecv error: recv got EOF from 127.0.0.1:49480 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 05:09:14.439224 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790226527740525-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.440071 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:14.440879 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:09:14.447296 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:14.441328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.0.98.1:43291)
0924 05:09:14.441364 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:14.441365 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:14.441426 (+ 61us) server_negotiation.cc:407] Connection header received
0924 05:09:14.441471 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:14.441472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:14.441478 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:14.441540 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:14.441846 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.442271 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.443025 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:14.443193 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:14.443245 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:14.445166 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:14.445167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:14.445167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:14.445174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:14.446258 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.446593 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.446594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.446595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.446646 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:14.446842 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:14.446843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:14.446844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:14.446932 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:14.447025 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:14.447137 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 05:09:14.447206 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:09:14.447810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 6017
I20260924 05:09:14.452857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 6151
I20260924 05:09:14.457228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 6285
I20260924 05:09:14.461233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 5923
2026-09-24T05:09:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1831 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[6434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[6434](info): set up 2 sockets
Sep 24 05:09:14 dist-test-slave-mr2f krb5kdc[6434](info): commencing operation
krb5kdc: starting...
W20260924 05:09:15.483026 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:15 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 05:09:15.595700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:15.673439 6447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:15.673672 6447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:15.673709 6447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:15.674846 6447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:15.674918 6447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:15.674942 6447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:15.674968 6447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:15.674996 6447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:15.676434 6447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:15.676693 6447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:15.676928 6447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:15.681492 6452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:15.681541 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:15.681537 6453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:15.681711 6447 server_base.cc:1034] running on GCE node
I20260924 05:09:15.681913 6447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:15.682147 6447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:15.683386 6447 hybrid_clock.cc:648] HybridClock initialized: now 1790226555683368 us; error 34 us; skew 500 ppm
Sep 24 05:09:15 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:15.685782 6447 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 05:09:15.686180 6447 webserver.cc:460] Webserver started at http://127.0.98.62:40029/ using document root <none> and password file <none>
I20260924 05:09:15.686409 6447 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:15.686475 6447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:15.686621 6447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:15.687678 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "be3fb6c102d140b789c4055d9fdd2280"
format_stamp: "Formatted at 2026-09-24 05:09:15 on dist-test-slave-mr2f"
server_key: "249694619c651896edcd58eeadbbabef"
server_key_iv: "47b31f05313884d7b2c0a60863e20ea2"
server_key_version: "encryptionkey@0"
I20260924 05:09:15.688027 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "be3fb6c102d140b789c4055d9fdd2280"
format_stamp: "Formatted at 2026-09-24 05:09:15 on dist-test-slave-mr2f"
server_key: "249694619c651896edcd58eeadbbabef"
server_key_iv: "47b31f05313884d7b2c0a60863e20ea2"
server_key_version: "encryptionkey@0"
I20260924 05:09:15.689392 6447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:15.690042 6462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:15.690232 6447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:15.690330 6447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "be3fb6c102d140b789c4055d9fdd2280"
format_stamp: "Formatted at 2026-09-24 05:09:15 on dist-test-slave-mr2f"
server_key: "249694619c651896edcd58eeadbbabef"
server_key_iv: "47b31f05313884d7b2c0a60863e20ea2"
server_key_version: "encryptionkey@0"
I20260924 05:09:15.690409 6447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:15.702675 6447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:15.704342 6447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:15.704496 6447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:15.708200 6447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45509
I20260924 05:09:15.708235 6514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45509 every 8 connection(s)
I20260924 05:09:15.708568 6447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:15.709522 6515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:15.709678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 6447
I20260924 05:09:15.709786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:15.709982 392 external_mini_cluster.cc:1428] Setting key 0ebcbe4bb64f32bcc7e772c4879181c5
I20260924 05:09:15.712719 6515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: Bootstrap starting.
I20260924 05:09:15.713456 6515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:15.713738 6515 log.cc:824] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:15.714426 6515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: No bootstrap required, opened a new log
I20260924 05:09:15.716419 6515 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } }
I20260924 05:09:15.716547 6515 raft_consensus.cc:374] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 24 05:09:15 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:15.716571 6515 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3fb6c102d140b789c4055d9fdd2280, State: Initialized, Role: FOLLOWER
I20260924 05:09:15.716707 6515 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } }
I20260924 05:09:15.716792 6515 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:15.716816 6515 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:15.716885 6515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:15.717617 6515 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } }
I20260924 05:09:15.717727 6515 leader_election.cc:302] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3fb6c102d140b789c4055d9fdd2280; no voters:
I20260924 05:09:15.717886 6515 leader_election.cc:288] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:15.717919 6520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:15.718087 6520 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 1 LEADER]: Becoming Leader. State: Replica: be3fb6c102d140b789c4055d9fdd2280, State: Running, Role: LEADER
I20260924 05:09:15.718139 6515 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:15.718174 6520 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } }
I20260924 05:09:15.718518 6520 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3fb6c102d140b789c4055d9fdd2280. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } }
I20260924 05:09:15.718595 6520 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:15.718873 6528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:15.719519 6528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:15.719731 6521 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } }
I20260924 05:09:15.719924 6521 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:15.721256 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:15.710477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.0.98.62:45509)
0924 05:09:15.710646 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:15.710653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:15.710677 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:09:15.711117 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:15.711123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:15.711177 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:15.711285 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:15.712041 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:15.712941 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:15.714026 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:15.714528 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:15.714654 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:15.717148 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:15.717153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:15.717153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:15.717164 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:15.719266 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:15.719710 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:15.719714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:15.719714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:15.719821 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:15.720071 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:15.720072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:15.720073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:15.720380 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:15.720547 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:15.720924 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 05:09:15.721048 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260924 05:09:15.721390 6528 catalog_manager.cc:1347] Generated new cluster ID: 03e5c7555e154f9095a9d3b8698904d0
I20260924 05:09:15.721424 6528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:15.734483 6528 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:15.735266 6528 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:15.739823 6528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: Generated new TSK 0
I20260924 05:09:15.739993 6528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 05:09:15.771311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:15.851841 6541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:15.852046 6541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:15.852068 6541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:15.853237 6541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:15.853279 6541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:15.853307 6541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:15.854736 6541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:15.855055 6541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:15.855304 6541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:15.860311 6547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:15.860316 6549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:15.860332 6541 server_base.cc:1034] running on GCE node
W20260924 05:09:15.860332 6546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:15.860649 6541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:15.860854 6541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:15.862007 6541 hybrid_clock.cc:648] HybridClock initialized: now 1790226555861993 us; error 28 us; skew 500 ppm
Sep 24 05:09:15 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:15.864523 6541 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 05:09:15.864981 6541 webserver.cc:460] Webserver started at http://127.0.98.1:42293/ using document root <none> and password file <none>
I20260924 05:09:15.865187 6541 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:15.865228 6541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:15.865316 6541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:15.866408 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "6c91da51a17b439d811e79b4f59f500e"
format_stamp: "Formatted at 2026-09-24 05:09:15 on dist-test-slave-mr2f"
server_key: "86e9749e552aed47d75081ace3cb5d5c"
server_key_iv: "67a73a9c40841a5802e70598c3742808"
server_key_version: "encryptionkey@0"
I20260924 05:09:15.866814 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6c91da51a17b439d811e79b4f59f500e"
format_stamp: "Formatted at 2026-09-24 05:09:15 on dist-test-slave-mr2f"
server_key: "86e9749e552aed47d75081ace3cb5d5c"
server_key_iv: "67a73a9c40841a5802e70598c3742808"
server_key_version: "encryptionkey@0"
I20260924 05:09:15.868144 6541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:15.868942 6556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:15.869148 6541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:15.869277 6541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "6c91da51a17b439d811e79b4f59f500e"
format_stamp: "Formatted at 2026-09-24 05:09:15 on dist-test-slave-mr2f"
server_key: "86e9749e552aed47d75081ace3cb5d5c"
server_key_iv: "67a73a9c40841a5802e70598c3742808"
server_key_version: "encryptionkey@0"
I20260924 05:09:15.869357 6541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:15.887338 6541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:15.889742 6541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:15.889909 6541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:15.890154 6541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:15.890480 6541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:15.890537 6541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:15.890582 6541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:15.890604 6541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:15.896909 6541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43613
I20260924 05:09:15.896952 6669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43613 every 8 connection(s)
I20260924 05:09:15.897291 6541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:09:15 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:15.905880 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:15.898037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42411 (local address 127.0.98.62:45509)
0924 05:09:15.898083 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:15.898085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:15.898282 (+ 197us) server_negotiation.cc:407] Connection header received
0924 05:09:15.898810 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:15.898811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:15.898823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:15.898890 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:15.899788 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:15.900300 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:15.901257 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:15.901445 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:15.901496 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:15.903559 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:15.903561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:15.903561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:15.903568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:15.904734 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:15.905085 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:15.905086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:15.905086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:15.905131 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:15.905347 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:15.905348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:15.905348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:15.905458 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:15.905507 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:15.905730 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 05:09:15.905810 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:09:15.905970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 6541
I20260924 05:09:15.906064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:15.906129 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:15.898016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:42411)
0924 05:09:15.898158 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:15.898194 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:15.898680 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:15.898952 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:15.898956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:15.899508 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:15.899702 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:15.899707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:15.900381 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:15.900382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:15.901184 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:15.901186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:15.901516 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:15.901517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:15.901616 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:15.902071 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:15.902079 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:15.903476 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:15.904806 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:15.904812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:15.904813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:15.905021 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:15.905201 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:15.905201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:15.905201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:15.905294 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:15.905511 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:15.905515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:15.905623 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 05:09:15.905711 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 05:09:15.905872 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260924 05:09:15.906292 392 external_mini_cluster.cc:1428] Setting key acc35eb47f00c76dfd7aab86c9e17776
I20260924 05:09:15.906706 6670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45509
I20260924 05:09:15.906826 6670 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:15.907012 6670 heartbeater.cc:511] Master 127.0.98.62:45509 requested a full tablet report, sending...
I20260924 05:09:15.907583 6479 ts_manager.cc:194] Registered new tserver with Master: 6c91da51a17b439d811e79b4f59f500e (127.0.98.1:43613)
I20260924 05:09:15.908177 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42411
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 05:09:15.929555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:16.010352 6675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:16.010562 6675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:16.010586 6675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:16.011747 6675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:16.011790 6675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:16.011816 6675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:16.013195 6675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:16.013384 6675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:16.013602 6675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:16.018556 6683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.018675 6675 server_base.cc:1034] running on GCE node
W20260924 05:09:16.018570 6681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.018570 6680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.019091 6675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:16.019309 6675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:16.020442 6675 hybrid_clock.cc:648] HybridClock initialized: now 1790226556020430 us; error 29 us; skew 500 ppm
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:16.022737 6675 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 05:09:16.023202 6675 webserver.cc:460] Webserver started at http://127.0.98.2:38429/ using document root <none> and password file <none>
I20260924 05:09:16.023432 6675 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:16.023500 6675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:16.023648 6675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:16.024675 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "dc364bc179174f3ab1fa7f7f1503bc2d"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "3bd2dab43574888c92b73e451d89bafb"
server_key_iv: "4990f42de5d9f4d9ea761241be5183aa"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.025035 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dc364bc179174f3ab1fa7f7f1503bc2d"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "3bd2dab43574888c92b73e451d89bafb"
server_key_iv: "4990f42de5d9f4d9ea761241be5183aa"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.026301 6675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:09:16.027046 6690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.027200 6675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.027298 6675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "dc364bc179174f3ab1fa7f7f1503bc2d"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "3bd2dab43574888c92b73e451d89bafb"
server_key_iv: "4990f42de5d9f4d9ea761241be5183aa"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.027385 6675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:16.053987 6675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:16.055841 6675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:16.056001 6675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:16.056232 6675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:16.056540 6675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:16.056595 6675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.056632 6675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:16.056660 6675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.062318 6675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34269
I20260924 05:09:16.062397 6803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34269 every 8 connection(s)
I20260924 05:09:16.062714 6675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:16.064246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 6675
I20260924 05:09:16.064370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:16.064549 392 external_mini_cluster.cc:1428] Setting key 11f8f09e1f5ea2a6b89d146f37a390d1
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 05:09:16.073233 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.063449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55643 (local address 127.0.98.62:45509)
0924 05:09:16.063513 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.063515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.063709 (+ 194us) server_negotiation.cc:407] Connection header received
0924 05:09:16.064219 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.064220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.064233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.064288 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.065503 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.066005 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.067507 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.067698 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.067760 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.070173 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.070174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.070175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.070183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.071687 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.072177 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.072179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.072179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.072244 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.072505 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.072507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.072507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.072655 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.072741 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.073011 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.073141 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:09:16.073402 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.063429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:55643)
0924 05:09:16.063571 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.063611 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.064112 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.064374 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.064378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.065105 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.065411 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.065416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.066101 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.066103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.067405 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.067408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.067788 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.067789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.067939 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.068436 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.068446 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.070070 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.071794 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.071800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.071802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.072102 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.072320 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.072321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.072321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.072433 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.072733 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.072739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.072878 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.072989 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.073147 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260924 05:09:16.073870 6804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45509
I20260924 05:09:16.073979 6804 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:16.074162 6804 heartbeater.cc:511] Master 127.0.98.62:45509 requested a full tablet report, sending...
I20260924 05:09:16.074575 6479 ts_manager.cc:194] Registered new tserver with Master: dc364bc179174f3ab1fa7f7f1503bc2d (127.0.98.2:34269)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 05:09:16.075088 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 05:09:16.089890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:16.171790 6809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:16.172035 6809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:16.172075 6809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:16.173370 6809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:16.173439 6809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:16.173482 6809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:16.174969 6809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:16.175241 6809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:16.175473 6809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:16.180728 6817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.180825 6809 server_base.cc:1034] running on GCE node
W20260924 05:09:16.180857 6814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.180977 6815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.181172 6809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:16.181385 6809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:16.182533 6809 hybrid_clock.cc:648] HybridClock initialized: now 1790226556182520 us; error 25 us; skew 500 ppm
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:16.189630 6809 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 05:09:16.190073 6809 webserver.cc:460] Webserver started at http://127.0.98.3:39951/ using document root <none> and password file <none>
I20260924 05:09:16.190321 6809 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:16.190394 6809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:16.190547 6809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:16.191676 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "6a0f16121973dea1ef600c010bc33414"
server_key_iv: "59f1e37461ce2293325b8eb4dba26043"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.192044 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "6a0f16121973dea1ef600c010bc33414"
server_key_iv: "59f1e37461ce2293325b8eb4dba26043"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.193478 6809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:16.194285 6824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.194468 6809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:16.194567 6809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "6a0f16121973dea1ef600c010bc33414"
server_key_iv: "59f1e37461ce2293325b8eb4dba26043"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.194653 6809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:16.208940 6809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:16.211367 6809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:16.211534 6809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:16.211772 6809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:16.212097 6809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:16.212157 6809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.212226 6809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:16.212258 6809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.218236 6809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46515
I20260924 05:09:16.218297 6937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46515 every 8 connection(s)
I20260924 05:09:16.218636 6809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:16.226297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 6809
I20260924 05:09:16.226367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:16.226533 392 external_mini_cluster.cc:1428] Setting key 40253c383359f48bc54a262b21e91e3e
I20260924 05:09:16.227917 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.219477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52371 (local address 127.0.98.62:45509)
0924 05:09:16.219521 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.219524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.219774 (+ 250us) server_negotiation.cc:407] Connection header received
0924 05:09:16.220347 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.220348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.220358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.220409 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.221325 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.221810 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.222840 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.223040 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.223105 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.225202 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.225203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.225203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.225209 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.226415 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.226941 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.226943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.226943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.227006 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.227265 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.227266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.227266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.227384 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.227461 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.227739 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.227817 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:16.228101 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.219487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:52371)
0924 05:09:16.219639 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.219677 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.220247 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.220473 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.220476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.221047 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.221244 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.221248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.221899 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.221899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.222692 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.222694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.223120 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.223121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.223221 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.223753 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.223762 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.225117 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.226519 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.226525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.226528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.226858 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.227081 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.227082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.227082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.227192 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.227451 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.227458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.227601 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.227705 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.227863 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260924 05:09:16.228574 6938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45509
I20260924 05:09:16.228677 6938 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:16.228852 6938 heartbeater.cc:511] Master 127.0.98.62:45509 requested a full tablet report, sending...
I20260924 05:09:16.229233 6479 ts_manager.cc:194] Registered new tserver with Master: be1f6210a1d247a49dbf6a7f5ec5a5b0 (127.0.98.3:46515)
I20260924 05:09:16.229599 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52371
I20260924 05:09:16.238705 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 05:09:16.263860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39249
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45509,127.0.98.61:39249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:16.342456 6943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:16.342656 6943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:16.342679 6943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:16.343828 6943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:16.343871 6943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:16.343885 6943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:16.343899 6943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:16.343953 6943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:16.345359 6943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45509,127.0.98.61:39249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
W20260924 05:09:16.345480 6943 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45509,127.0.98.61:39249'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 05:09:16.345628 6943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:16.345853 6943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:16.350540 6951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.350549 6949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.350659 6943 server_base.cc:1034] running on GCE node
W20260924 05:09:16.350549 6948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.351029 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:16.351265 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:16.352402 6943 hybrid_clock.cc:648] HybridClock initialized: now 1790226556352395 us; error 34 us; skew 500 ppm
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:16.354841 6943 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 05:09:16.355274 6943 webserver.cc:460] Webserver started at http://127.0.98.61:33131/ using document root <none> and password file <none>
I20260924 05:09:16.355448 6943 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:16.355486 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:16.355571 6943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:16.356556 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data/instance:
uuid: "c60180f652c7462c890b08111d966b47"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "f87bc55524e1a0c601ee96eba56619f1"
server_key_iv: "5dab0e71f9343206608430684de66af6"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.356859 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal/instance:
uuid: "c60180f652c7462c890b08111d966b47"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "f87bc55524e1a0c601ee96eba56619f1"
server_key_iv: "5dab0e71f9343206608430684de66af6"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.358112 6943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:16.358753 6958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.358990 6943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:16.359090 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal
uuid: "c60180f652c7462c890b08111d966b47"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "f87bc55524e1a0c601ee96eba56619f1"
server_key_iv: "5dab0e71f9343206608430684de66af6"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.359176 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:16.371181 6943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:16.373185 6943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:16.373335 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:16.377188 6943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39249
I20260924 05:09:16.377224 7010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39249 every 8 connection(s)
I20260924 05:09:16.377555 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:09:16.377804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 6943
I20260924 05:09:16.378655 7011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:16.380502 7011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:16.388679 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.381176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.0.98.62:45509)
0924 05:09:16.381248 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.381250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.381393 (+ 143us) server_negotiation.cc:407] Connection header received
0924 05:09:16.381451 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.381452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.381460 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.381511 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.382023 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.382485 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.383591 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.383799 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.383885 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.385970 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.385971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.385971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.385980 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.387377 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.387792 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.387793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.387793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.387860 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.388038 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.388039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.388039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.388230 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.388294 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.388501 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.388574 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 05:09:16.388803 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.378452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.98.61:39249)
0924 05:09:16.378641 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.378651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.378684 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:09:16.379345 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.379353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.379407 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.379574 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.380165 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.381146 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.382242 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.382641 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.382757 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.385294 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.385299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.385300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.385305 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.387140 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.387506 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.387508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.387509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.387581 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.387791 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.387792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.387792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.388123 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.388256 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.388376 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.388503 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260924 05:09:16.388808 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.381118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47146)
0924 05:09:16.381285 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.381326 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.381393 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.381545 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.381548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.381864 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.381951 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.381953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.382594 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.382595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.383485 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.383487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.383891 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.383892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.383965 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.384424 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.384431 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.385879 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.387449 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.387455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.387458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.387696 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.387924 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.387924 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.387925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.387975 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.388299 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.388306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.388369 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.388458 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.388575 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":16,"thread_start_us":100,"threads_started":1}
I20260924 05:09:16.389669 7011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 05:09:16.395746 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.389964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.0.1:37474)
0924 05:09:16.390011 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.390022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.390058 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.390173 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.390173 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.390274 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.390336 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.390338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.391013 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.391014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.391485 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.391486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.391824 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.391825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.391874 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.392266 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.392270 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.393396 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.394878 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.394879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.394879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.395034 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.395217 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.395218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.395218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.395265 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.395485 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.395485 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.395520 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.395569 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.395663 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:09:16.395797 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.390010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.0.98.61:39249)
0924 05:09:16.390038 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.390040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.390047 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:09:16.390113 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.390113 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.390120 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.390171 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.390409 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.390952 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.391555 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.391763 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.391816 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.393466 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.393467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.393467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.393472 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.394819 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.395110 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.395110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.395111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.395153 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.395333 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.395333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.395333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.395429 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.395478 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.395589 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.395706 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":13}
I20260924 05:09:16.397043 7011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: Bootstrap starting.
I20260924 05:09:16.398007 7011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:16.398285 7011 log.cc:824] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:16.399091 7011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: No bootstrap required, opened a new log
I20260924 05:09:16.400580 7011 raft_consensus.cc:348] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } }
I20260924 05:09:16.400768 7011 raft_consensus.cc:374] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:16.400816 7011 raft_consensus.cc:729] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60180f652c7462c890b08111d966b47, State: Initialized, Role: FOLLOWER
I20260924 05:09:16.400944 7011 consensus_queue.cc:260] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } }
I20260924 05:09:16.401294 7011 sys_catalog.cc:564] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:16.401300 7017 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } } }
I20260924 05:09:16.401518 7017 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:09:16.403393 7028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:09:16.403470 7028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:09:16.405133 6943 master_runner.cc:428] Detected that this master c60180f652c7462c890b08111d966b47 is joining an existing cluster
I20260924 05:09:16.405198 6943 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:39249
I20260924 05:09:16.413931 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.407131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47156)
0924 05:09:16.407500 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.407512 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.407577 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.407749 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.407750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.407970 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.408039 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.408041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.408743 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.408745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.409595 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.409598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.410091 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.410092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.410142 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.410604 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.410610 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.411091 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.412807 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.412808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.412809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.412981 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.413236 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.413236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.413237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.413302 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.413622 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.413622 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.413664 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.413730 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.413831 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":89,"threads_started":1}
I20260924 05:09:16.413965 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.407369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.0.98.62:45509)
0924 05:09:16.407424 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.407427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.407569 (+ 142us) server_negotiation.cc:407] Connection header received
0924 05:09:16.407637 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.407638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.407647 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.407715 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.408115 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.408660 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.409696 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.410012 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.410093 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.411197 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.411198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.411199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.411206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.412707 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.413101 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.413102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.413103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.413170 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.413395 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.413396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.413397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.413515 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.413550 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.413788 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.413869 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:09:16.417402 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.407376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.0.1:37488)
0924 05:09:16.410908 (+ 3532us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.410919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.410983 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.411136 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.411136 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.411329 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.411387 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.411388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.412109 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.412110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.412950 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.412952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.413364 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.413364 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.413441 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.414437 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.414441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.414848 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.416375 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.416375 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.416376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.416577 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.416783 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.416784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.416784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.416836 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.417093 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.417094 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.417128 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.417195 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.417286 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3514,"thread_start_us":64,"threads_started":1}
I20260924 05:09:16.417445 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.407436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.0.98.61:39249)
0924 05:09:16.407474 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.407475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.410990 (+ 3515us) server_negotiation.cc:407] Connection header received
0924 05:09:16.411040 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.411041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.411052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.411118 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.411485 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.412027 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.413031 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.413287 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.413355 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.414950 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.414951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.414951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.414958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.416262 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.416664 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.416665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.416665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.416714 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.416919 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.416920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.416921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.417032 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.417102 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.417216 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.417333 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":20}
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 05:09:16.424567 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.416317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.0.1:37490)
0924 05:09:16.416484 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.416521 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.416584 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.417845 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.417856 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.418195 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.418240 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.418242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.418978 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.418979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.419638 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.419640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.420083 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.420084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.420162 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.420626 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.420648 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.421925 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.423577 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.423583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.423585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.423813 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.423998 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.423999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.423999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.424049 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.424279 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.424284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.424322 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.424390 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.424491 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":54,"threads_started":1}
I20260924 05:09:16.424600 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.416384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.0.98.61:39249)
0924 05:09:16.417724 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.417726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.417735 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:16.417765 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.417766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.417775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.417825 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.418322 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.418897 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.419736 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.420002 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.420081 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.422004 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.422006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.422006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.422013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.423492 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.423880 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.423881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.423882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.423936 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.424105 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.424106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.424106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.424219 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.424283 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.424441 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.424512 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1325,"thread_start_us":86,"threads_started":1}
I20260924 05:09:16.425127 6479 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:39249: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c60180f652c7462c890b08111d966b47"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 39249
}
attrs {
promote: true
}
}
dest_uuid: "be3fb6c102d140b789c4055d9fdd2280"
cas_config_opid_index: -1
I20260924 05:09:16.425463 6479 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } }
I20260924 05:09:16.425973 7038 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c60180f652c7462c890b08111d966b47 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } } }
I20260924 05:09:16.426260 6985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:16.427165 7038 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:39249
I20260924 05:09:16.427174 6985 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Refusing update from remote peer be3fb6c102d140b789c4055d9fdd2280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 05:09:16.427379 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } } }
I20260924 05:09:16.427484 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:16.427485 6943 master.cc:537] Master@127.0.98.61:39249 shutting down...
I20260924 05:09:16.427450 7038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 05:09:16.428544 6463 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 05:09:16.429075 6463 consensus_peers.cc:487] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 -> Peer c60180f652c7462c890b08111d966b47 (127.0.98.61:39249): Couldn't send request to peer c60180f652c7462c890b08111d966b47. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 05:09:16.430154 6943 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:16.430236 6943 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:16.430302 6943 tablet_replica.cc:330] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: stopping tablet replica
I20260924 05:09:16.442703 6943 master.cc:552] Master@127.0.98.61:39249 shutdown complete.
I20260924 05:09:16.442749 6943 master_runner.cc:305] Clearing existing system tablet
I20260924 05:09:16.442992 6943 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:16.443032 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:16.443657 6943 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:16.444168 7043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.444286 6943 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:16.444348 6943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.444386 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal
uuid: "c60180f652c7462c890b08111d966b47"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "f87bc55524e1a0c601ee96eba56619f1"
server_key_iv: "5dab0e71f9343206608430684de66af6"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.445372 6943 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:16.446748 6943 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 05:09:16.446856 6943 log.cc:1196] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 05:09:16.447081 6943 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: Deleting consensus metadata
I20260924 05:09:16.447207 6943 master_runner.cc:315] Copying system tablet from 127.0.98.62:45509
I20260924 05:09:16.447836 6943 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45509
I20260924 05:09:16.454242 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.448158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47170)
0924 05:09:16.448260 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.448270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.448339 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.448488 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.448488 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.448639 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.448695 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.448696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.449637 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.449638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.450197 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.450198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.450678 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.450679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.450727 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.451312 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.451317 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.451612 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.453319 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.453320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.453320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.453487 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.453689 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.453690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.453690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.453740 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.454000 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.454001 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.454037 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.454089 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.454167 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":64,"threads_started":1}
I20260924 05:09:16.454299 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.448222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.0.98.62:45509)
0924 05:09:16.448268 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.448270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.448338 (+ 68us) server_negotiation.cc:407] Connection header received
0924 05:09:16.448406 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.448406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.448418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.448478 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.448763 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.449559 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.450287 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.450601 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.450675 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.451696 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.451698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.451698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.451706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.453236 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.453560 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.453562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.453562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.453625 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.453806 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.453807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.453808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.453939 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.454008 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.454130 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.454205 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:09:16.454571 6499 tablet_copy_service.cc:140] P be3fb6c102d140b789c4055d9fdd2280: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c60180f652c7462c890b08111d966b47 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47170)
I20260924 05:09:16.454671 6499 tablet_copy_service.cc:161] P be3fb6c102d140b789c4055d9fdd2280: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c60180f652c7462c890b08111d966b47 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47170: session id = c60180f652c7462c890b08111d966b47-00000000000000000000000000000000
I20260924 05:09:16.455300 6499 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 05:09:16.455915 6943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 05:09:16.456838 6943 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: tablet copy: Starting download of 0 data blocks...
I20260924 05:09:16.456992 6943 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: tablet copy: Starting download of 1 WAL segments...
I20260924 05:09:16.457804 6943 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 05:09:16.458698 6499 tablet_copy_service.cc:342] P be3fb6c102d140b789c4055d9fdd2280: Request end of tablet copy session c60180f652c7462c890b08111d966b47-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47170
I20260924 05:09:16.458766 6499 tablet_copy_service.cc:433] P be3fb6c102d140b789c4055d9fdd2280: ending tablet copy session c60180f652c7462c890b08111d966b47-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c60180f652c7462c890b08111d966b47
I20260924 05:09:16.461059 6943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:16.462236 7050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.462265 7051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.462363 7053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.462441 6943 server_base.cc:1034] running on GCE node
I20260924 05:09:16.462598 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:16.462725 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:16.463876 6943 hybrid_clock.cc:648] HybridClock initialized: now 1790226556463870 us; error 35 us; skew 500 ppm
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:16.465164 6943 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 05:09:16.465456 6943 webserver.cc:460] Webserver started at http://127.0.98.61:35987/ using document root <none> and password file <none>
I20260924 05:09:16.465588 6943 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:16.465629 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:16.466249 6943 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:16.466792 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.466889 6943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:16.466933 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal
uuid: "c60180f652c7462c890b08111d966b47"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "f87bc55524e1a0c601ee96eba56619f1"
server_key_iv: "5dab0e71f9343206608430684de66af6"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.466987 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:16.477742 6943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:16.479418 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:16.482487 6943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39249
I20260924 05:09:16.482678 7112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39249 every 8 connection(s)
I20260924 05:09:16.482913 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:09:16.483626 7113 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:09:16.484045 7113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: Bootstrap starting.
I20260924 05:09:16.486156 7113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:09:16.486392 7113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: Bootstrap complete.
I20260924 05:09:16.486584 7113 raft_consensus.cc:348] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } }
I20260924 05:09:16.486660 7113 raft_consensus.cc:729] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c60180f652c7462c890b08111d966b47, State: Initialized, Role: LEARNER
I20260924 05:09:16.486686 7113 consensus_queue.cc:260] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } }
I20260924 05:09:16.486881 7113 sys_catalog.cc:564] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:16.486878 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } } }
I20260924 05:09:16.487010 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: This master's current role is: LEARNER
I20260924 05:09:16.488324 7127 catalog_manager.cc:1259] Loaded cluster ID: 03e5c7555e154f9095a9d3b8698904d0
I20260924 05:09:16.488368 7127 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: loading cluster ID for follower catalog manager: success
I20260924 05:09:16.489264 7127 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: acquiring CA information for follower catalog manager: success
I20260924 05:09:16.489529 7127 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:09:16.498570 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.490456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.0.98.61:39249)
0924 05:09:16.490562 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.490564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.490572 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:16.490591 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.490592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.490600 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.490668 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.491386 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.491989 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.493050 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.493237 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.493289 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.494918 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.494919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.494919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.494925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.497421 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.497857 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.497857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.497858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.497913 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.498123 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.498123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.498123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.498218 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.498290 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.498438 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.498495 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 05:09:16.525607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39493
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45509,127.0.98.61:39249,127.0.98.60:39493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:16.606568 7131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:16.606781 7131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:16.606803 7131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:16.607954 7131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:16.607995 7131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:16.608011 7131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:16.608022 7131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:16.608088 7131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:16.609606 7131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45509,127.0.98.61:39249,127.0.98.60:39493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:16.609853 7131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:16.610087 7131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:16.614634 7137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.614637 7136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.614660 7139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.614989 7131 server_base.cc:1034] running on GCE node
I20260924 05:09:16.615221 7131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:16.615515 7131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:16.616700 7131 hybrid_clock.cc:648] HybridClock initialized: now 1790226556616695 us; error 32 us; skew 500 ppm
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:16.619474 7131 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 05:09:16.619899 7131 webserver.cc:460] Webserver started at http://127.0.98.60:40157/ using document root <none> and password file <none>
I20260924 05:09:16.620146 7131 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:16.620211 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:16.620358 7131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:16.621361 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data/instance:
uuid: "339dac30bfc541828c4365d10e80a0ff"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "0b3de54061a056be99ee12351f735cf0"
server_key_iv: "c831312364e7cf7ae0a342f75c9450a3"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.621708 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal/instance:
uuid: "339dac30bfc541828c4365d10e80a0ff"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "0b3de54061a056be99ee12351f735cf0"
server_key_iv: "c831312364e7cf7ae0a342f75c9450a3"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.623041 7131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:16.623723 7146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.623885 7131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:16.623994 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal
uuid: "339dac30bfc541828c4365d10e80a0ff"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "0b3de54061a056be99ee12351f735cf0"
server_key_iv: "c831312364e7cf7ae0a342f75c9450a3"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.624073 7131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:16.636909 7131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:16.639186 7131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:16.639338 7131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:16.642910 7131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39493
I20260924 05:09:16.642947 7198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39493 every 8 connection(s)
I20260924 05:09:16.643311 7131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:09:16.644224 7199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:16.645714 7199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:09:16.650269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 7131
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:16.654932 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.646533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.0.98.62:45509)
0924 05:09:16.646578 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.646580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.646640 (+ 60us) server_negotiation.cc:407] Connection header received
0924 05:09:16.647109 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.647110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.647119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.647181 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.647741 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.648319 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.649263 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.649518 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.649573 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.652055 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.652057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.652057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.652065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.653516 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.653957 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.653959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.653959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.654019 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.654289 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.654290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.654290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.654401 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.654474 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.654741 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.654839 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:09:16.655118 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.646376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47182)
0924 05:09:16.646529 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.646569 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.647032 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.647228 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.647230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.647488 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.647672 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.647675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.648402 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.648403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.649164 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.649166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.649589 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.649590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.649691 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.650145 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.650156 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.651959 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.653621 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.653627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.653629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.653871 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.654087 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.654088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.654088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.654206 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.654476 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.654482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.654614 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.654721 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.654893 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:09:16.655717 7199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:09:16.659071 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.650992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.0.98.60:39493)
0924 05:09:16.651128 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.651133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.651154 (+ 21us) server_negotiation.cc:407] Connection header received
0924 05:09:16.651172 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.651174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.651189 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.651269 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.651915 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.652642 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.653338 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.653634 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.653724 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.656116 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.656122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.656122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.656127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.657845 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.658220 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.658223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.658223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.658277 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.658462 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.658462 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.658463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.658615 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.658813 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.658878 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.658959 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 05:09:16.662340 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.656190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.0.1:37500)
0924 05:09:16.656243 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.656253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.656286 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.656429 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.656430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.656563 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.656611 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.656614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.657365 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.657366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.658105 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.658107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.658585 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.658586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.658657 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.659119 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.659124 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.660319 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.661611 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.661612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.661612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.661763 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.661927 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.661927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.661928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.661974 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.662139 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.662140 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.662169 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.662218 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.662285 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 05:09:16.662361 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.656273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.0.98.61:39249)
0924 05:09:16.656322 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.656325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.656334 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:16.656358 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.656359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.656373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.656426 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.656695 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.657304 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.658182 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.658511 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.658569 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.660396 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.660397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.660397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.660403 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.661544 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.661825 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.661825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.661826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.661872 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.662011 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.662011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.662012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.662093 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.662136 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.662239 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.662302 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:09:16.662732 7199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:09:16.668551 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.663000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.0.1:43636)
0924 05:09:16.663066 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.663083 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.663124 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.663208 (+ 84us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.663208 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.663316 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.663342 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.663344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.663870 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.663871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.664324 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.664325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.664612 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.664613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.664646 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.664940 (+ 294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.664943 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.665977 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.667499 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.667500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.667501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.667657 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.667893 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.667894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.667894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.667965 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.668197 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.668197 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.668232 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.668333 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.668424 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 05:09:16.668568 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.663048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.0.98.60:39493)
0924 05:09:16.663078 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.663080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.663114 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:09:16.663159 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.663160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.663167 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.663205 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.663396 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.663806 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.664388 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.664554 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.664601 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.666055 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.666056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.666056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.666062 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.667410 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.667740 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.667741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.667742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.667811 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.668028 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.668029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.668029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.668127 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.668203 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.668354 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.668473 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":15}
I20260924 05:09:16.669988 7199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Bootstrap starting.
I20260924 05:09:16.670841 7199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:16.671473 7199 log.cc:824] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:16.672176 7199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: No bootstrap required, opened a new log
I20260924 05:09:16.673592 7199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } }
I20260924 05:09:16.673717 7199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:16.673739 7199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339dac30bfc541828c4365d10e80a0ff, State: Initialized, Role: FOLLOWER
I20260924 05:09:16.673833 7199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } }
I20260924 05:09:16.674053 7204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } } }
I20260924 05:09:16.674108 7199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:16.674129 7204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:09:16.675657 7215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:09:16.675711 7215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:09:16.677923 7131 master_runner.cc:428] Detected that this master 339dac30bfc541828c4365d10e80a0ff is joining an existing cluster
I20260924 05:09:16.677973 7131 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39493
I20260924 05:09:16.686590 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.679773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.98.61:39249)
0924 05:09:16.679813 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.679815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.680455 (+ 640us) server_negotiation.cc:407] Connection header received
0924 05:09:16.680546 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.680547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.680556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.680586 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.681023 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.681764 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.682886 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.683174 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.683214 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.684200 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.684201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.684201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.684207 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.685572 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.685948 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.685949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.685949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.685995 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.686179 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.686179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.686179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.686268 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.686316 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.686471 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.686527 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:09:16.686648 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.679634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.0.1:37508)
0924 05:09:16.680406 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.680415 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.680478 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.680642 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.680643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.680867 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.680934 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.680936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.681869 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.681870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.682754 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.682794 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.683305 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.683306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.683371 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.683695 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.683700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.684095 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.685655 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.685656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.685656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.685872 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.686055 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.686055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.686056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.686119 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.686332 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.686333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.686376 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.686456 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.686546 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":57,"threads_started":1}
I20260924 05:09:16.706044 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.679632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.0.98.60:39493)
0924 05:09:16.679673 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.679675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.679697 (+ 22us) server_negotiation.cc:407] Connection header received
0924 05:09:16.679718 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.679720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.679729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.679775 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.680138 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.680814 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.681806 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.682091 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.682126 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.703301 (+ 21175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.703302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.703303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.703308 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.704844 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.705197 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.705198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.705198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.705254 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.705451 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.705452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.705452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.705584 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.705616 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.705844 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.705922 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:09:16.706374 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.679474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.0.98.62:45509)
0924 05:09:16.679520 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.679522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.679819 (+ 297us) server_negotiation.cc:407] Connection header received
0924 05:09:16.679916 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.679917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.679927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.679957 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.680379 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.681073 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.682047 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.682325 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.682365 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.703291 (+ 20926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.703294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.703294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.703304 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.705169 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.705525 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.705526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.705526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.705585 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.705805 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.705806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.705806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.705946 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.705990 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.706214 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.706288 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:09:16.706388 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.679414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.0.1:43652)
0924 05:09:16.679582 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.679594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.679652 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.679818 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.679818 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.679992 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.680055 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.680057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.680906 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.680907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.681726 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.681728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.682196 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.682196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.682245 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.682710 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.682716 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.703164 (+ 20448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.704947 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.704947 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.704948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.705118 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.705321 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.705322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.705322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.705384 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.705675 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.705676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.705737 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.706181 (+ 444us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.706278 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260924 05:09:16.707099 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.679414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47198)
0924 05:09:16.679766 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.679776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.679845 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.680017 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.680017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.680210 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.680286 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.680288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.681152 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.681153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.681968 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.681969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.682792 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.682793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.682859 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.683280 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.683284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.703164 (+ 19880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.705248 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.705248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.705249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.705452 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.705659 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.705659 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.705660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.705716 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.706037 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.706038 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.706094 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.706457 (+ 363us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.706974 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260924 05:09:16.710213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 6541
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226555, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:09:16.715936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43613
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45509
--webserver_port=42293
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45509,127.0.98.61:39249
--tserver_master_addrs=127.0.98.62:45509,127.0.98.61:39249,127.0.98.60:39493 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:09:16.717949 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.709684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.0.98.60:39493)
0924 05:09:16.709716 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.709718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.709724 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:09:16.709753 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.709753 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.709762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.710107 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.710144 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.711679 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.712698 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.712985 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.713287 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.715184 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.715185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.715186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.715192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.716776 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.717193 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.717195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.717195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.717251 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.717459 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.717460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.717461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.717569 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.717641 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.717789 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.717861 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:16.717955 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.709496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.0.1:43666)
0924 05:09:16.709569 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.709578 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.709612 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.709836 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.709837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.709996 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.710041 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.710043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.711779 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.711780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.712604 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.712607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.713065 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.713066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.713130 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.713553 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.713558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.715100 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.716874 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.716875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.716876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.717100 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.717329 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.717330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.717330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.717388 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.717635 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.717636 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.717677 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.717766 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.717861 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 05:09:16.718915 6479 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39493: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "339dac30bfc541828c4365d10e80a0ff"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39493
}
attrs {
promote: true
}
}
dest_uuid: "be3fb6c102d140b789c4055d9fdd2280"
cas_config_opid_index: 5
I20260924 05:09:16.719161 6479 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } }
I20260924 05:09:16.719559 7039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 339dac30bfc541828c4365d10e80a0ff (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.720458 7039 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39493
I20260924 05:09:16.720533 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.720628 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:16.720997 7131 master.cc:537] Master@127.0.98.60:39493 shutting down...
I20260924 05:09:16.721148 7173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:16.721946 7173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Refusing update from remote peer be3fb6c102d140b789c4055d9fdd2280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 05:09:16.722798 7038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 05:09:16.723482 6464 consensus_peers.cc:487] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 -> Peer 339dac30bfc541828c4365d10e80a0ff (127.0.98.60:39493): Couldn't send request to peer 339dac30bfc541828c4365d10e80a0ff. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 05:09:16.724988 7131 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:16.725077 7131 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:16.725126 7131 tablet_replica.cc:330] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: stopping tablet replica
I20260924 05:09:16.728091 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.720769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.0.98.61:39249)
0924 05:09:16.720813 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.720815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.724629 (+ 3814us) server_negotiation.cc:407] Connection header received
0924 05:09:16.724750 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.724751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.724775 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.724830 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:16.725106 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.725813 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.727368 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.727756 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.727847 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.727914 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.727997 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.728031 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:09:16.728092 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.719594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.0.1:37522)
0924 05:09:16.724450 (+ 4856us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.724480 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.724663 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.724904 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.724905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.724907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:16.725018 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.725020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.725924 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.725925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.727277 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.727279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.727844 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.727845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.727911 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.727932 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.727990 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.728028 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4840}
I20260924 05:09:16.728348 7087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 LEARNER]: Refusing update from remote peer be3fb6c102d140b789c4055d9fdd2280: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 05:09:16.728531 7038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 05:09:16.729121 7087 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 339dac30bfc541828c4365d10e80a0ff (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.730512 7038 raft_consensus.cc:1050] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: attempting to promote NON_VOTER c60180f652c7462c890b08111d966b47 to VOTER
I20260924 05:09:16.730545 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3fb6c102d140b789c4055d9fdd2280. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.730616 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: This master's current role is: LEARNER
I20260924 05:09:16.730648 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: true } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.730683 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: This master's current role is: LEARNER
I20260924 05:09:16.730707 7038 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } }
I20260924 05:09:16.731277 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.731180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.0.1:43670)
0924 05:09:16.731211 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.731249 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 05:09:16.731420 7087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 LEARNER]: Refusing update from remote peer be3fb6c102d140b789c4055d9fdd2280: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 05:09:16.731792 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.731690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.0.1:43684)
0924 05:09:16.731746 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.731773 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 05:09:16.731755 7227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 05:09:16.731953 6464 consensus_peers.cc:487] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 -> Peer 339dac30bfc541828c4365d10e80a0ff (127.0.98.60:39493): Couldn't send request to peer 339dac30bfc541828c4365d10e80a0ff. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39493: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 05:09:16.732448 7039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c60180f652c7462c890b08111d966b47 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.733163 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.733239 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:16.733295 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.733351 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:16.733548 7087 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c60180f652c7462c890b08111d966b47 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.734297 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.734387 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:09:16.737919 7131 master.cc:552] Master@127.0.98.60:39493 shutdown complete.
I20260924 05:09:16.737982 7131 master_runner.cc:305] Clearing existing system tablet
I20260924 05:09:16.738112 7131 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:16.738168 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:16.738890 7131 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:16.739348 7229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.739475 7131 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:16.739521 7131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.739548 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal
uuid: "339dac30bfc541828c4365d10e80a0ff"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "0b3de54061a056be99ee12351f735cf0"
server_key_iv: "c831312364e7cf7ae0a342f75c9450a3"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.740149 7131 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:16.741403 7131 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 05:09:16.741478 7131 log.cc:1196] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 05:09:16.741674 7131 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Deleting consensus metadata
I20260924 05:09:16.741817 7131 master_runner.cc:315] Copying system tablet from 127.0.98.62:45509
I20260924 05:09:16.742403 7131 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45509
I20260924 05:09:16.753098 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.742750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.0.98.62:45509)
0924 05:09:16.742835 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.742837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.747017 (+ 4180us) server_negotiation.cc:407] Connection header received
0924 05:09:16.747121 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.747123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.747136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.747170 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.747576 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.748083 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.748815 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.749032 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.749107 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.750451 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.750452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.750453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.750460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.751967 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.752304 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.752305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.752306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.752362 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.752562 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.752563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.752564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.752672 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.752749 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.752915 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.752984 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 05:09:16.753141 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.742706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47206)
0924 05:09:16.746935 (+ 4229us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.746946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.747032 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.747283 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.747284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.747447 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.747491 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.747493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.748171 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.748172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.748716 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.748718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.749105 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.749106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.749162 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.750032 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.750036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.750326 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.752060 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.752061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.752062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.752234 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.752432 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.752433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.752433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.752490 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.752740 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.752741 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.752787 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.752869 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.752982 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4200,"thread_start_us":94,"threads_started":1}
I20260924 05:09:16.753760 6499 tablet_copy_service.cc:140] P be3fb6c102d140b789c4055d9fdd2280: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 339dac30bfc541828c4365d10e80a0ff ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47206)
I20260924 05:09:16.753834 6499 tablet_copy_service.cc:161] P be3fb6c102d140b789c4055d9fdd2280: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 339dac30bfc541828c4365d10e80a0ff at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47206: session id = 339dac30bfc541828c4365d10e80a0ff-00000000000000000000000000000000
I20260924 05:09:16.754281 6499 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 05:09:16.754822 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 05:09:16.755769 7131 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: tablet copy: Starting download of 0 data blocks...
I20260924 05:09:16.755944 7131 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: tablet copy: Starting download of 1 WAL segments...
I20260924 05:09:16.756605 7131 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 05:09:16.757323 6499 tablet_copy_service.cc:342] P be3fb6c102d140b789c4055d9fdd2280: Request end of tablet copy session 339dac30bfc541828c4365d10e80a0ff-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47206
I20260924 05:09:16.757375 6499 tablet_copy_service.cc:433] P be3fb6c102d140b789c4055d9fdd2280: ending tablet copy session 339dac30bfc541828c4365d10e80a0ff-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 339dac30bfc541828c4365d10e80a0ff
I20260924 05:09:16.759707 7131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:16.760900 7236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.760921 7237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.761225 7131 server_base.cc:1034] running on GCE node
W20260924 05:09:16.766403 7239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.766577 7131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:16.766755 7131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:16.767987 7131 hybrid_clock.cc:648] HybridClock initialized: now 1790226556767978 us; error 27 us; skew 500 ppm
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:16.769306 7131 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 05:09:16.769589 7131 webserver.cc:460] Webserver started at http://127.0.98.60:34713/ using document root <none> and password file <none>
I20260924 05:09:16.769706 7131 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:16.769740 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:16.770332 7131 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:16.770890 7246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.771031 7131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:16.771075 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal
uuid: "339dac30bfc541828c4365d10e80a0ff"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "0b3de54061a056be99ee12351f735cf0"
server_key_iv: "c831312364e7cf7ae0a342f75c9450a3"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.771135 7131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:16.781419 7131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:16.783099 7131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:16.786096 7131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39493
I20260924 05:09:16.786238 7298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39493 every 8 connection(s)
I20260924 05:09:16.786434 7131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:09:16.787292 7299 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:09:16.787811 7299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Bootstrap starting.
I20260924 05:09:16.790162 7299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:09:16.790402 7299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Bootstrap complete.
I20260924 05:09:16.790620 7299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } }
I20260924 05:09:16.790710 7299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 339dac30bfc541828c4365d10e80a0ff, State: Initialized, Role: LEARNER
I20260924 05:09:16.790755 7299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } }
I20260924 05:09:16.790930 7299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:16.790926 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:16.791132 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: This master's current role is: LEARNER
I20260924 05:09:16.792565 7313 catalog_manager.cc:1259] Loaded cluster ID: 03e5c7555e154f9095a9d3b8698904d0
I20260924 05:09:16.792747 7313 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: loading cluster ID for follower catalog manager: success
I20260924 05:09:16.793694 7313 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: acquiring CA information for follower catalog manager: success
I20260924 05:09:16.793989 7313 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 05:09:16.832610 7224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:16.832891 7224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:16.832928 7224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:16.834537 7224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:16.834597 7224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:16.834636 7224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:16.836786 7224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42293
--tserver_master_addrs=127.0.98.62:45509,127.0.98.61:39249,127.0.98.60:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:16.837024 7224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:16.837244 7224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:16.842334 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.842253 7319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:16.842249 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:16.842286 7224 server_base.cc:1034] running on GCE node
I20260924 05:09:16.842695 7224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:16.842983 7224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:16.844159 7224 hybrid_clock.cc:648] HybridClock initialized: now 1790226556844138 us; error 34 us; skew 500 ppm
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:16.847031 7224 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 05:09:16.847473 7224 webserver.cc:460] Webserver started at http://127.0.98.1:42293/ using document root <none> and password file <none>
I20260924 05:09:16.847705 7224 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:16.847788 7224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:16.849256 7224 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:16.849907 7328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.850078 7224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:16.850171 7224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "6c91da51a17b439d811e79b4f59f500e"
format_stamp: "Formatted at 2026-09-24 05:09:15 on dist-test-slave-mr2f"
server_key: "86e9749e552aed47d75081ace3cb5d5c"
server_key_iv: "67a73a9c40841a5802e70598c3742808"
server_key_version: "encryptionkey@0"
I20260924 05:09:16.850518 7224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:16.885468 7224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:16.888090 7224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:16.888259 7224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:16.888494 7224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:16.888799 7224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:16.888852 7224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.888940 7224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:16.888970 7224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:16.895610 7224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43613
I20260924 05:09:16.896092 7441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43613 every 8 connection(s)
I20260924 05:09:16.896423 7224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:16.903479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 7224
I20260924 05:09:16.903645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 6675
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 05:09:16 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226556, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:09:16.908079 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.897229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55753 (local address 127.0.98.62:45509)
0924 05:09:16.897271 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.897273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.897682 (+ 409us) server_negotiation.cc:407] Connection header received
0924 05:09:16.898619 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.898621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.898636 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.898697 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.899528 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.900035 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.901031 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.901237 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.901343 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.903945 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.903946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.903947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.903954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.905614 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.906074 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.906075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.906076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.906134 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.906941 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.906942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.906942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.907090 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.907138 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.907879 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.907961 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:09:16.908674 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.897529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44533 (local address 127.0.98.61:39249)
0924 05:09:16.897587 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.897589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.897744 (+ 155us) server_negotiation.cc:407] Connection header received
0924 05:09:16.898689 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.898690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.898702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.898734 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.899544 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.900283 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.901336 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.901626 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.901663 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.905357 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.905359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.905359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.905367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.907159 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.907554 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.907555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.907555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.907613 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.907814 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.907815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.907815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.907969 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.908009 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.908453 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.908538 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:09:16.908686 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.897114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:55753)
0924 05:09:16.897615 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.897624 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.898489 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.898820 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.898824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.899237 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.899434 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.899438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.900142 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.900142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.900930 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.900932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.901349 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.901351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.901472 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.902094 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.902105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.903833 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.905698 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.905705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.905707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.905977 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.906445 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.906446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.906446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.906578 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.907191 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.907197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.907618 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.908188 (+ 570us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.908416 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260924 05:09:16.909397 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.897409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.98.1:44533)
0924 05:09:16.897705 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.897712 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.898496 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.898819 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.898824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.899266 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.899436 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.899439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.900372 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.900372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.901243 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.901245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.901734 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.901735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.901796 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.902129 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.902133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.905266 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.907250 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.907251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.907251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.907471 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.907685 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.907686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.907686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.907751 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.908054 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.908055 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.908099 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.909141 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.909281 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":25,"thread_start_us":67,"threads_started":1}
I20260924 05:09:16.909482 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.897104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45793 (local address 127.0.98.60:39493)
0924 05:09:16.897235 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:16.897237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:16.898624 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 05:09:16.898659 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:16.898660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:16.898699 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:16.898793 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:16.899545 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.900527 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.901459 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.901761 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.901827 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.906394 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:16.906395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:16.906395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:16.906403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:16.908080 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.908498 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.908499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.908499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.908554 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:16.908907 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:16.908908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:16.908908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:16.909008 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:16.909102 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:16.909289 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 05:09:16.909369 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260924 05:09:16.909497 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:16.897114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.98.1:45793)
0924 05:09:16.897375 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:16.897419 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:16.898484 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:16.898819 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:16.898824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:16.899277 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:16.899436 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.899441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.900610 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.900611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.901383 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:16.901384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:16.901838 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:16.901839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:16.901920 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:16.902305 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:16.902309 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:16.906303 (+ 3994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:16.908164 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.908164 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.908165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.908418 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.908616 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:16.908617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:16.908617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:16.908667 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:16.909096 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:16.909097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:16.909156 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 05:09:16.909263 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 05:09:16.909363 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260924 05:09:16.910272 7444 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39249
I20260924 05:09:16.910360 7444 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:16.910501 7444 heartbeater.cc:511] Master 127.0.98.61:39249 requested a full tablet report, sending...
I20260924 05:09:16.910738 7443 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39493
I20260924 05:09:16.910807 7443 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:16.910876 7443 heartbeater.cc:511] Master 127.0.98.60:39493 requested a full tablet report, sending...
I20260924 05:09:16.910992 7077 ts_manager.cc:194] Registered new tserver with Master: 6c91da51a17b439d811e79b4f59f500e (127.0.98.1:43613)
I20260924 05:09:16.911243 7263 ts_manager.cc:194] Registered new tserver with Master: 6c91da51a17b439d811e79b4f59f500e (127.0.98.1:43613)
I20260924 05:09:16.911275 7442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45509
I20260924 05:09:16.911310 7442 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:16.911366 7442 heartbeater.cc:511] Master 127.0.98.62:45509 requested a full tablet report, sending...
I20260924 05:09:16.911602 6479 ts_manager.cc:194] Re-registered known tserver with Master: 6c91da51a17b439d811e79b4f59f500e (127.0.98.1:43613)
I20260924 05:09:16.911873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34269
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45509
--webserver_port=38429
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45509,127.0.98.61:39249
--tserver_master_addrs=127.0.98.62:45509,127.0.98.61:39249,127.0.98.60:39493 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:09:16.912056 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55753
W20260924 05:09:16.994170 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:16.994392 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:16.994429 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:16.995625 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:16.995724 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:16.995770 7450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:16.997241 7450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38429
--tserver_master_addrs=127.0.98.62:45509,127.0.98.61:39249,127.0.98.60:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:16.997498 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:16.997748 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:17.002707 7456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:17.002722 7450 server_base.cc:1034] running on GCE node
W20260924 05:09:17.002707 7455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:17.002782 7458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:17.003170 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:17.003407 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:17.004545 7450 hybrid_clock.cc:648] HybridClock initialized: now 1790226557004528 us; error 39 us; skew 500 ppm
Sep 24 05:09:17 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:17.006980 7450 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 05:09:17.007344 7450 webserver.cc:460] Webserver started at http://127.0.98.2:38429/ using document root <none> and password file <none>
I20260924 05:09:17.007512 7450 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:17.007561 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:17.008805 7450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:17.009322 7465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:17.009500 7450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:17.009580 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "dc364bc179174f3ab1fa7f7f1503bc2d"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "3bd2dab43574888c92b73e451d89bafb"
server_key_iv: "4990f42de5d9f4d9ea761241be5183aa"
server_key_version: "encryptionkey@0"
I20260924 05:09:17.009888 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:17.032011 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:17.033924 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:17.034034 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:17.034221 7450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:17.034468 7450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:17.034498 7450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:17.034519 7450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:17.034533 7450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:17.040169 7450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34269
I20260924 05:09:17.040228 7578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34269 every 8 connection(s)
I20260924 05:09:17.040571 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:17.046906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 7450
I20260924 05:09:17.047060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 6809
Sep 24 05:09:17 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 05:09:17 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 05:09:17 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 05:09:17.052795 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.041777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39931 (local address 127.0.98.61:39249)
0924 05:09:17.041818 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.041820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.041829 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:17.042389 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.042390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.042399 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.042461 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.043113 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.043598 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.044986 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.045170 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.045223 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.049362 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.049363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.049364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.049371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.051183 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.051677 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.051678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.051679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.051735 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.052054 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.052055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.052056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.052195 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.052241 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.052628 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.052700 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:09:17.052800 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.041258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45561 (local address 127.0.98.60:39493)
0924 05:09:17.041649 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.041651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.041661 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:17.042275 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.042276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.042287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.042339 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.043135 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.043896 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.045803 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.046126 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.046228 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.049038 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.049039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.049040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.049047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.051055 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.051677 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.051678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.051679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.051735 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.052054 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.052055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.052055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.052196 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.052249 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.052613 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.052716 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20260924 05:09:17.052821 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.041259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59463 (local address 127.0.98.62:45509)
0924 05:09:17.041506 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.041508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.041593 (+ 85us) server_negotiation.cc:407] Connection header received
0924 05:09:17.042210 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.042211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.042223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.042278 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.043130 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.043901 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.045269 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.045449 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.045527 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.048459 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.048460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.048461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.048468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.051303 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.051771 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.051772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.051772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.051831 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.052043 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.052044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.052045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.052198 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.052241 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.052584 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.052671 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260924 05:09:17.053433 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.041542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.98.2:39931)
0924 05:09:17.041763 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.041771 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.042113 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.042479 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.042480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.042806 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:17.043018 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.043023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.043700 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.043701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.044902 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.044904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.045247 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.045248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.045378 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.045948 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:17.045956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:17.049293 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:17.051263 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.051263 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.051264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.051586 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.051813 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.051813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.051814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.051972 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.052288 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:17.052288 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:17.052437 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.052901 (+ 464us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.053138 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260924 05:09:17.053438 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.041264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:59463)
0924 05:09:17.041459 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.041507 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.042096 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.042719 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.042720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.042919 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:17.043020 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.043026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.043975 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.043976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.044902 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.044904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.045528 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.045529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.045609 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.046125 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:17.046130 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:17.048365 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:17.051391 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.051391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.051392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.051689 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.051899 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.051900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.051900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.051977 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.052285 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:17.052286 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:17.052438 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.052932 (+ 494us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.053233 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":147,"spinlock_wait_cycles":7424,"thread_start_us":96,"threads_started":1}
I20260924 05:09:17.054049 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.041264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.98.2:45561)
0924 05:09:17.041459 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.041507 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.042096 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.042368 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.042373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.042828 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:17.043020 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.043025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.045001 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.045001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.045734 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.045737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.046231 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.046233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.046303 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.046708 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:17.046713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:17.048931 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:17.051150 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.051157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.051159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.051586 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.051813 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.051813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.051814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.051972 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.052274 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:17.052280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:17.052437 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.053826 (+ 1389us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.053910 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260924 05:09:17.054520 7580 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39493
I20260924 05:09:17.054620 7580 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:17.054745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46515
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45509
--webserver_port=39951
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45509,127.0.98.61:39249
--tserver_master_addrs=127.0.98.62:45509,127.0.98.61:39249,127.0.98.60:39493 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:09:17.054842 7580 heartbeater.cc:511] Master 127.0.98.60:39493 requested a full tablet report, sending...
I20260924 05:09:17.055117 7579 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45509
I20260924 05:09:17.055117 7581 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39249
I20260924 05:09:17.055188 7579 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:17.055202 7581 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:17.055263 7579 heartbeater.cc:511] Master 127.0.98.62:45509 requested a full tablet report, sending...
I20260924 05:09:17.055266 7581 heartbeater.cc:511] Master 127.0.98.61:39249 requested a full tablet report, sending...
I20260924 05:09:17.055558 6479 ts_manager.cc:194] Re-registered known tserver with Master: dc364bc179174f3ab1fa7f7f1503bc2d (127.0.98.2:34269)
I20260924 05:09:17.055693 7077 ts_manager.cc:194] Registered new tserver with Master: dc364bc179174f3ab1fa7f7f1503bc2d (127.0.98.2:34269)
I20260924 05:09:17.056092 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59463
I20260924 05:09:17.056464 7263 ts_manager.cc:194] Registered new tserver with Master: dc364bc179174f3ab1fa7f7f1503bc2d (127.0.98.2:34269)
W20260924 05:09:17.137671 7586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:17.137887 7586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:17.137909 7586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:17.139098 7586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:17.139138 7586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:17.139165 7586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:17.140677 7586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39951
--tserver_master_addrs=127.0.98.62:45509,127.0.98.61:39249,127.0.98.60:39493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:17.140926 7586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:17.141170 7586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:17.146132 7592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:17.146145 7594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:17.146193 7591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:17.146376 7586 server_base.cc:1034] running on GCE node
I20260924 05:09:17.146749 7586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:17.147058 7586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:17.148218 7586 hybrid_clock.cc:648] HybridClock initialized: now 1790226557148208 us; error 27 us; skew 500 ppm
Sep 24 05:09:17 dist-test-slave-mr2f krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:17.150861 7586 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 05:09:17.151307 7586 webserver.cc:460] Webserver started at http://127.0.98.3:39951/ using document root <none> and password file <none>
I20260924 05:09:17.151500 7586 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:17.151551 7586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:17.152796 7586 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:17.153367 7601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:17.153555 7586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:09:17.153653 7586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0"
format_stamp: "Formatted at 2026-09-24 05:09:16 on dist-test-slave-mr2f"
server_key: "6a0f16121973dea1ef600c010bc33414"
server_key_iv: "59f1e37461ce2293325b8eb4dba26043"
server_key_version: "encryptionkey@0"
I20260924 05:09:17.154017 7586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:17.166886 7586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:17.169243 7586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:17.169402 7586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:17.169631 7586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:17.169927 7586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:17.169981 7586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:17.170022 7586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:17.170049 7586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:17.176147 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.173506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.0.98.60:39493)
0924 05:09:17.173559 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.173562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.173571 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:17.173642 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.173643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.173664 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.173750 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:17.173890 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.174375 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.175364 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.175808 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.175896 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.175968 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.176039 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.176075 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:09:17.176148 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.173463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.0.1:43688)
0924 05:09:17.173517 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.173528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.173574 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.173757 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.173758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.173760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:17.173810 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.173812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.174471 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.174472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.175274 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.175276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.175903 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.175905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.175967 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.175971 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.176032 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.176080 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 05:09:17.176424 7273 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 05:09:17.176596 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3fb6c102d140b789c4055d9fdd2280. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:17.176713 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: This master's current role is: LEARNER
I20260924 05:09:17.176710 7586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46515
I20260924 05:09:17.177124 7586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:17.177568 7714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46515 every 8 connection(s)
I20260924 05:09:17.179926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 7586
Sep 24 05:09:17 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 05:09:17 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 05:09:17.190171 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.178299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37075 (local address 127.0.98.60:39493)
0924 05:09:17.178364 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.178366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.178466 (+ 100us) server_negotiation.cc:407] Connection header received
0924 05:09:17.178830 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.178831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.178841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.178875 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.179869 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.180602 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.182288 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.182577 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.182635 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.186401 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.186403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.186403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.186410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.188369 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.188945 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.188946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.188947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.189003 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.189285 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.189287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.189287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.189411 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.189476 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.189949 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.190056 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 05:09:17 dist-test-slave-mr2f krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:17.190670 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.178101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.98.3:37075)
0924 05:09:17.178398 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.178408 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.178709 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.178985 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.178986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.179506 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:17.179760 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.179766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.180752 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.180752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.182161 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.182164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.182688 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.182689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.182754 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.183564 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:17.183574 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:17.186312 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:17.188504 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.188511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.188514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.188825 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.189078 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.189080 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.189080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.189229 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.189479 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:17.189486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:17.189662 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.189780 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.190390 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":220,"thread_start_us":64,"threads_started":1}
I20260924 05:09:17.191363 7716 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39493
I20260924 05:09:17.191485 7716 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:17.191566 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.181681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.0.98.60:39493)
0924 05:09:17.183037 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.183039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.183165 (+ 126us) server_negotiation.cc:407] Connection header received
0924 05:09:17.183236 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.183237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.183248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.183333 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.185055 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.185524 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.186533 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.186886 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.186969 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.188471 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.188472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.188473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.188479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.190168 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.190578 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.190579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.190579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.190629 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.190863 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.190864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.190864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.191003 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.191095 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.191396 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.191473 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1339,"thread_start_us":60,"threads_started":1}
I20260924 05:09:17.191701 7716 heartbeater.cc:511] Master 127.0.98.60:39493 requested a full tablet report, sending...
I20260924 05:09:17.192166 7263 ts_manager.cc:194] Registered new tserver with Master: be1f6210a1d247a49dbf6a7f5ec5a5b0 (127.0.98.3:46515)
I20260924 05:09:17.193403 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.178193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49979 (local address 127.0.98.61:39249)
0924 05:09:17.178241 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.178243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.178253 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:17.178830 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.178831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.178843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.178898 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.179895 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.180635 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.182281 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.182488 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.182573 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.186421 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.186423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.186424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.186432 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.189000 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.192503 (+ 3503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.192504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.192505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.192563 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.192784 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.192785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.192785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.192952 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.193017 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.193206 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.193294 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:09:17.193727 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.178077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.98.3:49979)
0924 05:09:17.178195 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.178206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.178702 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.179251 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.179252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.179514 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:17.179766 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.179770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.180742 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.180742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.182164 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.182166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.182584 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.182585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.182706 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.183567 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:17.183578 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:17.186313 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:17.189080 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.189081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.189081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.192430 (+ 3349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.192656 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.192657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.192657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.192718 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.193048 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:17.193049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:17.193092 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.193490 (+ 398us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.193596 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":146,"thread_start_us":62,"threads_started":1}
I20260924 05:09:17.194440 7717 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39249
I20260924 05:09:17.194519 7717 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:17.194594 7717 heartbeater.cc:511] Master 127.0.98.61:39249 requested a full tablet report, sending...
I20260924 05:09:17.194953 7077 ts_manager.cc:194] Registered new tserver with Master: be1f6210a1d247a49dbf6a7f5ec5a5b0 (127.0.98.3:46515)
I20260924 05:09:17.195338 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.177820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:49665)
0924 05:09:17.177992 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.178038 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.178700 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.178971 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.178976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.179481 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:17.179760 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.179766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.180733 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.180734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.182164 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.182167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.182687 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.182688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.182754 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.183677 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:17.183682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:17.192069 (+ 8387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:17.193991 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.193991 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.193992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.194208 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.194429 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:17.194430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:17.194430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:17.194506 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:17.195059 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:17.195060 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:17.195105 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.195173 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.195269 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":145,"thread_start_us":70,"threads_started":1}
I20260924 05:09:17.195384 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.177784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49665 (local address 127.0.98.62:45509)
0924 05:09:17.178217 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.178219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.178228 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:17.178840 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.178841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.178852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.178881 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.179869 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.180610 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.182275 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.182578 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.182635 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.192152 (+ 9517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.192153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.192153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.192160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.193910 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.194283 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.194284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.194285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.194342 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.194584 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.194585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.194586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.194732 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.194827 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.195208 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.195288 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
I20260924 05:09:17.195817 7715 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45509
I20260924 05:09:17.195865 7715 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:17.195931 7715 heartbeater.cc:511] Master 127.0.98.62:45509 requested a full tablet report, sending...
I20260924 05:09:17.196147 6479 ts_manager.cc:194] Re-registered known tserver with Master: be1f6210a1d247a49dbf6a7f5ec5a5b0 (127.0.98.3:46515)
I20260924 05:09:17.196651 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49665
I20260924 05:09:17.196915 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.181148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.0.98.62:45509)
0924 05:09:17.183310 (+ 2162us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.183311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.183318 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:09:17.183339 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.183340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.183350 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.183419 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.184126 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.184883 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.189177 (+ 4294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.189520 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.189581 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.191489 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.191490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.191490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.191497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.194972 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.196028 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.196029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.196030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.196093 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.196363 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.196364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.196365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.196476 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.196561 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.196725 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.196811 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2139,"thread_start_us":82,"threads_started":1}
I20260924 05:09:17.197542 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.181465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.0.98.61:39249)
0924 05:09:17.182990 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.182992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.183930 (+ 938us) server_negotiation.cc:407] Connection header received
0924 05:09:17.184033 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.184034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.184046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.184083 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.186678 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.187879 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.189872 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.190212 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.190255 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.193925 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.193926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.193927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.193933 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.195709 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.196343 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.196345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.196345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.196398 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.196613 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.196614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.196614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.196739 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.196826 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.197356 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.197440 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":52,"threads_started":1}
I20260924 05:09:17.198342 6479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:17.199885 6479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:17.200593 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: true } } }
I20260924 05:09:17.200701 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:17.201884 7039 raft_consensus.cc:1050] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: attempting to promote NON_VOTER 339dac30bfc541828c4365d10e80a0ff to VOTER
I20260924 05:09:17.202014 7039 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } }
I20260924 05:09:17.202638 7272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 LEARNER]: Refusing update from remote peer be3fb6c102d140b789c4055d9fdd2280: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 05:09:17.202919 7087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Refusing update from remote peer be3fb6c102d140b789c4055d9fdd2280: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 05:09:17.202988 7227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 05:09:17.203186 7227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 05:09:17.203727 7038 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 339dac30bfc541828c4365d10e80a0ff (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:17.203933 7272 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 339dac30bfc541828c4365d10e80a0ff (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:17.204057 7087 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 339dac30bfc541828c4365d10e80a0ff (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:17.204670 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:17.204770 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:17.204828 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:17.204885 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:17.204838 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:17.204931 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:09:17.204922 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:17.204963 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:17.205046 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be3fb6c102d140b789c4055d9fdd2280" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:17.205116 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:09:17.209649 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.205631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.0.1:39130)
0924 05:09:17.205677 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.205687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.205734 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.206327 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.206329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.206330 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:17.206396 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.206398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.207700 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.207701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.208386 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.208387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.209400 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.209401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.209460 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.209464 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.209532 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.209570 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260924 05:09:17.209684 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.205746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.0.98.1:43613)
0924 05:09:17.206167 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.206172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.206181 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:17.206238 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.206240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.206256 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.206334 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:17.206460 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.207604 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.208465 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.209054 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.209125 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.209473 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.209550 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.209600 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":74,"threads_started":1}
I20260924 05:09:17.209837 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.205701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46515 (local address 127.0.0.1:39218)
0924 05:09:17.205817 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.205826 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.205862 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.206387 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.206388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.206390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:17.206426 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.206427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.207509 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.207510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.208684 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.208686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.209608 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.209609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.209672 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.209677 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.209722 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.209768 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":4,"thread_start_us":78,"threads_started":1}
I20260924 05:09:17.210043 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.205872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34269 (local address 127.0.0.1:46510)
0924 05:09:17.206004 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.206014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.206064 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.206338 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.206339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.206341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:17.206395 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.206396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.207676 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.207677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.208728 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.208730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.209554 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.209554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.209604 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.209608 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.209636 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.209675 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260924 05:09:17.210427 7376 tablet_service.cc:1467] Processing CreateTablet for tablet 37f1df23fbb64e938983f7a24825ef28 (DEFAULT_TABLE table=test-table [id=cce7155c8a72450d8ebd33b9fb3f2828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:17.210565 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.205957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.0.98.2:34269)
0924 05:09:17.206144 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.206149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.206160 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:09:17.206204 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.206206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.206224 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.206466 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:17.206521 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.207596 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.208822 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.209464 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.210165 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.210459 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.210481 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.210510 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260924 05:09:17.210875 7376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37f1df23fbb64e938983f7a24825ef28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:17.211321 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.206079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.0.98.3:46515)
0924 05:09:17.206217 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.206222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.206232 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:17.206267 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.206270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.206286 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.206494 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:17.206547 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.207409 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.208780 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.209497 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.211082 (+ 1585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.211157 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.211184 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.211243 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260924 05:09:17.211407 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 37f1df23fbb64e938983f7a24825ef28 (DEFAULT_TABLE table=test-table [id=cce7155c8a72450d8ebd33b9fb3f2828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:17.211760 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37f1df23fbb64e938983f7a24825ef28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:17.211956 7649 tablet_service.cc:1467] Processing CreateTablet for tablet 37f1df23fbb64e938983f7a24825ef28 (DEFAULT_TABLE table=test-table [id=cce7155c8a72450d8ebd33b9fb3f2828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:17.212257 7649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37f1df23fbb64e938983f7a24825ef28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:17.214118 7744 tablet_bootstrap.cc:492] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: Bootstrap starting.
I20260924 05:09:17.214720 7744 tablet_bootstrap.cc:654] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:17.215036 7744 log.cc:824] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:17.215276 7745 tablet_bootstrap.cc:492] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: Bootstrap starting.
I20260924 05:09:17.215955 7745 tablet_bootstrap.cc:654] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:17.216229 7744 tablet_bootstrap.cc:492] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: No bootstrap required, opened a new log
I20260924 05:09:17.216248 7745 log.cc:824] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:17.216297 7744 ts_tablet_manager.cc:1397] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:09:17.216858 7745 tablet_bootstrap.cc:492] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: No bootstrap required, opened a new log
I20260924 05:09:17.216950 7745 ts_tablet_manager.cc:1397] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:17.216990 7746 tablet_bootstrap.cc:492] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Bootstrap starting.
I20260924 05:09:17.217605 7746 tablet_bootstrap.cc:654] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:17.217904 7746 log.cc:824] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:17.218704 7746 tablet_bootstrap.cc:492] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: No bootstrap required, opened a new log
I20260924 05:09:17.218806 7746 ts_tablet_manager.cc:1397] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:09:17.218734 7745 raft_consensus.cc:348] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:17.218936 7745 raft_consensus.cc:374] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:17.218986 7745 raft_consensus.cc:729] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc364bc179174f3ab1fa7f7f1503bc2d, State: Initialized, Role: FOLLOWER
I20260924 05:09:17.218699 7744 raft_consensus.cc:348] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:17.219101 7745 consensus_queue.cc:260] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:17.219149 7744 raft_consensus.cc:374] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:17.219202 7744 raft_consensus.cc:729] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c91da51a17b439d811e79b4f59f500e, State: Initialized, Role: FOLLOWER
I20260924 05:09:17.219348 7745 ts_tablet_manager.cc:1428] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:09:17.219323 7744 consensus_queue.cc:260] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:17.219478 7579 heartbeater.cc:503] Master 127.0.98.62:45509 was elected leader, sending a full tablet report...
I20260924 05:09:17.219928 7744 ts_tablet_manager.cc:1428] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260924 05:09:17.220042 7442 heartbeater.cc:503] Master 127.0.98.62:45509 was elected leader, sending a full tablet report...
I20260924 05:09:17.221925 7746 raft_consensus.cc:348] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:17.222044 7746 raft_consensus.cc:374] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:17.222065 7746 raft_consensus.cc:729] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1f6210a1d247a49dbf6a7f5ec5a5b0, State: Initialized, Role: FOLLOWER
I20260924 05:09:17.222194 7746 consensus_queue.cc:260] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:17.222446 7746 ts_tablet_manager.cc:1428] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 05:09:17.222503 7715 heartbeater.cc:503] Master 127.0.98.62:45509 was elected leader, sending a full tablet report...
I20260924 05:09:17.225507 7750 raft_consensus.cc:482] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:17.225600 7750 raft_consensus.cc:504] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:17.225926 7750 leader_election.cc:288] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c91da51a17b439d811e79b4f59f500e (127.0.98.1:43613), be1f6210a1d247a49dbf6a7f5ec5a5b0 (127.0.98.3:46515)
I20260924 05:09:17.229727 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.226081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46515 (local address 127.0.98.2:38193)
0924 05:09:17.226138 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.226151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.226189 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.226316 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.226317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.226319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:17.226422 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.226423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.227335 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.227336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.228627 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.228629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.229407 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.229408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.229487 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.229494 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.229553 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.229600 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":88}
I20260924 05:09:17.229744 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.226137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38193 (local address 127.0.98.3:46515)
0924 05:09:17.226176 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.226178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.226187 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:17.226242 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.226243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.226252 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.226313 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:17.226492 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.227254 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.228762 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.229316 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.229398 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.229408 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.229636 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.229678 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:17.229736 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.226081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.98.2:35435)
0924 05:09:17.226138 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:17.226151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:17.226189 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:17.226318 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:17.226318 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:17.226319 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:17.226422 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.226423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.227335 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.227336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.228627 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:17.228629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.229407 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.229408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:17.229487 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.229494 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 05:09:17.229553 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:09:17.229600 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":126}
I20260924 05:09:17.229770 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.226137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35435 (local address 127.0.98.1:43613)
0924 05:09:17.226175 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.226178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.226187 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:17.226242 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.226243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.226252 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.226313 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:17.226493 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.227260 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.228762 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.229316 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.229398 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.229408 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.229636 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.229678 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:17.230084 7669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f1df23fbb64e938983f7a24825ef28" candidate_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" is_pre_election: true
I20260924 05:09:17.230208 7669 raft_consensus.cc:2454] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc364bc179174f3ab1fa7f7f1503bc2d in term 0.
I20260924 05:09:17.230296 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f1df23fbb64e938983f7a24825ef28" candidate_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c91da51a17b439d811e79b4f59f500e" is_pre_election: true
I20260924 05:09:17.230409 7469 leader_election.cc:302] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be1f6210a1d247a49dbf6a7f5ec5a5b0, dc364bc179174f3ab1fa7f7f1503bc2d; no voters:
I20260924 05:09:17.230469 7396 raft_consensus.cc:2454] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc364bc179174f3ab1fa7f7f1503bc2d in term 0.
I20260924 05:09:17.230561 7750 raft_consensus.cc:2781] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:17.230628 7750 raft_consensus.cc:482] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:17.230661 7750 raft_consensus.cc:3037] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:17.231411 7750 raft_consensus.cc:504] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:17.231571 7750 leader_election.cc:288] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [CANDIDATE]: Term 1 election: Requested vote from peers 6c91da51a17b439d811e79b4f59f500e (127.0.98.1:43613), be1f6210a1d247a49dbf6a7f5ec5a5b0 (127.0.98.3:46515)
I20260924 05:09:17.231727 7669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f1df23fbb64e938983f7a24825ef28" candidate_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0"
I20260924 05:09:17.231837 7669 raft_consensus.cc:3037] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:17.232486 7669 raft_consensus.cc:2454] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc364bc179174f3ab1fa7f7f1503bc2d in term 1.
I20260924 05:09:17.232576 7396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f1df23fbb64e938983f7a24825ef28" candidate_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c91da51a17b439d811e79b4f59f500e"
I20260924 05:09:17.232625 7396 raft_consensus.cc:3037] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:17.232758 7469 leader_election.cc:302] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be1f6210a1d247a49dbf6a7f5ec5a5b0, dc364bc179174f3ab1fa7f7f1503bc2d; no voters:
I20260924 05:09:17.232906 7750 raft_consensus.cc:2781] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:17.233050 7750 raft_consensus.cc:686] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 1 LEADER]: Becoming Leader. State: Replica: dc364bc179174f3ab1fa7f7f1503bc2d, State: Running, Role: LEADER
I20260924 05:09:17.233136 7750 consensus_queue.cc:237] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:17.233537 7396 raft_consensus.cc:2454] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc364bc179174f3ab1fa7f7f1503bc2d in term 1.
I20260924 05:09:17.234161 6479 catalog_manager.cc:5495] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d reported cstate change: term changed from 0 to 1, leader changed from <none> to dc364bc179174f3ab1fa7f7f1503bc2d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } health_report { overall_health: HEALTHY } } }
I20260924 05:09:17.246802 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.244665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.0.98.2:34269)
0924 05:09:17.244752 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.244754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.244762 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:17.244792 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.244793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.244810 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.244882 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:17.244995 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.245441 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.246117 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.246292 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.246340 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.246468 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:17.246560 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:17.246617 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.246647 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.246664 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:09:17.248594 7669 raft_consensus.cc:1261] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 1 FOLLOWER]: Refusing update from remote peer dc364bc179174f3ab1fa7f7f1503bc2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:17.248620 7396 raft_consensus.cc:1261] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [term 1 FOLLOWER]: Refusing update from remote peer dc364bc179174f3ab1fa7f7f1503bc2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:17.248977 7750 consensus_queue.cc:1035] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:17.249114 7753 consensus_queue.cc:1035] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:17.252641 7758 mvcc.cc:205] Tried to move back new op lower bound from 7332767978487435264 to 7332767978427645952. Current Snapshot: MvccSnapshot[applied={T|T < 7332767978487435264}]
I20260924 05:09:17.255354 6478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:17.255466 6478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:17.256894 6478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be3fb6c102d140b789c4055d9fdd2280: Sending DeleteTablet for 3 replicas of tablet 37f1df23fbb64e938983f7a24825ef28
I20260924 05:09:17.257205 7649 tablet_service.cc:1514] Processing DeleteTablet for tablet 37f1df23fbb64e938983f7a24825ef28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39218
I20260924 05:09:17.257324 7513 tablet_service.cc:1514] Processing DeleteTablet for tablet 37f1df23fbb64e938983f7a24825ef28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46510
I20260924 05:09:17.257452 7765 tablet_replica.cc:330] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: stopping tablet replica
I20260924 05:09:17.257545 7765 raft_consensus.cc:2229] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:17.257658 7765 raft_consensus.cc:2258] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:17.257783 7376 tablet_service.cc:1514] Processing DeleteTablet for tablet 37f1df23fbb64e938983f7a24825ef28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39130
I20260924 05:09:17.258103 7765 ts_tablet_manager.cc:1905] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:17.258376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 6447
I20260924 05:09:17.260880 7766 tablet_replica.cc:330] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: stopping tablet replica
I20260924 05:09:17.260983 7766 raft_consensus.cc:2229] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:17.261065 7766 raft_consensus.cc:2258] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:17.261200 7765 ts_tablet_manager.cc:1918] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:17.261265 7765 log.cc:1196] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/37f1df23fbb64e938983f7a24825ef28
I20260924 05:09:17.261551 7765 ts_tablet_manager.cc:1939] T 37f1df23fbb64e938983f7a24825ef28 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Deleting consensus metadata
I20260924 05:09:17.262305 7766 ts_tablet_manager.cc:1905] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:17.263974 7766 ts_tablet_manager.cc:1918] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:17.264056 7766 log.cc:1196] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/37f1df23fbb64e938983f7a24825ef28
I20260924 05:09:17.264328 7766 ts_tablet_manager.cc:1939] T 37f1df23fbb64e938983f7a24825ef28 P dc364bc179174f3ab1fa7f7f1503bc2d: Deleting consensus metadata
W20260924 05:09:17.264642 7604 connection.cc:535] server connection from 127.0.0.1:39218 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 05:09:17.264653 7469 connection.cc:441] server connection from 127.0.0.1:46510 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46510 (request call id 1) could send its response
I20260924 05:09:17.264820 7768 tablet_replica.cc:330] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: stopping tablet replica
I20260924 05:09:17.264933 7768 raft_consensus.cc:2229] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:17.265043 7768 raft_consensus.cc:2258] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:17.265538 7768 ts_tablet_manager.cc:1905] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:17.267357 7768 ts_tablet_manager.cc:1918] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:17.267446 7768 log.cc:1196] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/37f1df23fbb64e938983f7a24825ef28
I20260924 05:09:17.267706 7768 ts_tablet_manager.cc:1939] T 37f1df23fbb64e938983f7a24825ef28 P 6c91da51a17b439d811e79b4f59f500e: Deleting consensus metadata
W20260924 05:09:17.268077 7331 connection.cc:441] server connection from 127.0.0.1:39130 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39130 (request call id 1) could send its response
I20260924 05:09:17.269328 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.259287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.0.98.60:39493)
0924 05:09:17.259327 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.259329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.259442 (+ 113us) server_negotiation.cc:407] Connection header received
0924 05:09:17.259537 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.259538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.259549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.259579 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.260439 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.261198 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.262511 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.262812 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.262857 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.264892 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.264893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.264894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.264901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.267971 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.268406 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.268407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.268407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.268470 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.268724 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.268725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.268725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.268870 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.268973 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.269146 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.269238 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:17.275290 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.268418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.98.61:39249)
0924 05:09:17.268460 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.268462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.268644 (+ 182us) server_negotiation.cc:407] Connection header received
0924 05:09:17.268684 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.268684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.268699 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.268747 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.269463 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.270022 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.270838 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.271071 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.271162 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.272524 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.272526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.272526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.272533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.273953 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.274337 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.274338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.274339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.274396 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.274593 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.274594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.274594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.274704 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.274825 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.275091 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.275174 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:09:17.281394 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:17.275004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.0.98.60:39493)
0924 05:09:17.275042 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:17.275043 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:17.275205 (+ 162us) server_negotiation.cc:407] Connection header received
0924 05:09:17.275307 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:17.275308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:17.275320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:17.275372 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:17.276126 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.276590 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.277275 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:17.277479 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:17.277586 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:17.278777 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:17.278778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:17.278779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:17.278786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:17.280290 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.280631 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.280632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.280633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.280684 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:17.280890 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:17.280891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:17.280891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:17.281000 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:17.281112 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:17.281260 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 05:09:17.281331 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:09:18.253901 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.253509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:47495)
0924 05:09:18.253644 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.253756 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
W20260924 05:09:18.254205 7442 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45509 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:18.254278 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.254017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:46871)
0924 05:09:18.254143 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.254237 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":46,"threads_started":1}
W20260924 05:09:18.254544 7715 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45509 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:18.254647 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.254444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:55041)
0924 05:09:18.254540 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.254604 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":44,"threads_started":1}
W20260924 05:09:18.254873 7579 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45509 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:18.757112 7778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be3fb6c102d140b789c4055d9fdd2280)
I20260924 05:09:18.757236 7778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } }
I20260924 05:09:18.757498 7778 leader_election.cc:288] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509), c60180f652c7462c890b08111d966b47 (127.0.98.61:39249)
I20260924 05:09:18.757862 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.757668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47674)
0924 05:09:18.757765 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.757814 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":53,"threads_started":1}
I20260924 05:09:18.757891 7780 raft_consensus.cc:482] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be3fb6c102d140b789c4055d9fdd2280)
W20260924 05:09:18.758008 7247 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:18.758057 7780 raft_consensus.cc:504] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } }
I20260924 05:09:18.758340 7780 leader_election.cc:288] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509), 339dac30bfc541828c4365d10e80a0ff (127.0.98.60:39493)
I20260924 05:09:18.758558 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.758393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47690)
0924 05:09:18.758472 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.758523 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":47,"threads_started":1}
W20260924 05:09:18.758652 7061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:18.758802 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.758260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47678)
0924 05:09:18.758318 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.758749 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 05:09:18.758876 7247 leader_election.cc:334] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:18.759058 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.758978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47702)
0924 05:09:18.759013 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.759034 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 05:09:18.759136 7061 leader_election.cc:334] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:18.761382 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.757825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39249 (local address 127.0.0.1:37544)
0924 05:09:18.757920 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.757930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:18.758209 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:18.758440 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:18.758441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:18.758444 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:18.758503 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:18.758505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:18.759456 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:18.759456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:18.760550 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:18.760553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:18.761149 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:18.761149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:18.761204 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:18.761208 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:09:18.761251 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:09:18.761284 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
I20260924 05:09:18.761370 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.757900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.0.98.61:39249)
0924 05:09:18.758046 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:18.758049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:18.758271 (+ 222us) server_negotiation.cc:407] Connection header received
0924 05:09:18.758302 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:18.758303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:18.758314 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:18.758371 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:18.758584 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:18.759369 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:18.760633 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:18.761051 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:18.761141 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:18.761152 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:18.761273 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 05:09:18.761310 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260924 05:09:18.761814 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.758582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.0.98.60:39493)
0924 05:09:18.758909 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:18.758911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:18.758920 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:18.758943 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:18.758944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:18.758955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:18.759058 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:18.759193 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:18.759747 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:18.760935 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:18.761535 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:18.761601 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:18.761611 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:18.761725 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 05:09:18.761761 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":79,"threads_started":1}
I20260924 05:09:18.761790 7087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "339dac30bfc541828c4365d10e80a0ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c60180f652c7462c890b08111d966b47" is_pre_election: true
I20260924 05:09:18.761881 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.758470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39493 (local address 127.0.0.1:43750)
0924 05:09:18.758606 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.758616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:18.758657 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:18.759068 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:18.759069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:18.759071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:18.759113 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:18.759115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:18.759822 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:18.759823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:18.760858 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:18.760860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:18.761602 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:18.761603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:18.761657 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:18.761660 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:18.761706 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 05:09:18.761764 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":41,"threads_started":1}
I20260924 05:09:18.761895 7087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 339dac30bfc541828c4365d10e80a0ff in term 1.
I20260924 05:09:18.762106 7247 leader_election.cc:302] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 339dac30bfc541828c4365d10e80a0ff, c60180f652c7462c890b08111d966b47; no voters: be3fb6c102d140b789c4055d9fdd2280
I20260924 05:09:18.762293 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c60180f652c7462c890b08111d966b47" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "339dac30bfc541828c4365d10e80a0ff" is_pre_election: true
I20260924 05:09:18.762396 7272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60180f652c7462c890b08111d966b47 in term 1.
I20260924 05:09:18.762490 7778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 05:09:18.762522 7778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader be3fb6c102d140b789c4055d9fdd2280)
I20260924 05:09:18.762544 7778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:09:18.762569 7062 leader_election.cc:302] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 339dac30bfc541828c4365d10e80a0ff, c60180f652c7462c890b08111d966b47; no voters: be3fb6c102d140b789c4055d9fdd2280
I20260924 05:09:18.762810 7780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 05:09:18.762859 7780 raft_consensus.cc:482] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader be3fb6c102d140b789c4055d9fdd2280)
I20260924 05:09:18.762878 7780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:09:18.763363 7778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } }
I20260924 05:09:18.763509 7778 leader_election.cc:288] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 2 election: Requested vote from peers be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509), c60180f652c7462c890b08111d966b47 (127.0.98.61:39249)
I20260924 05:09:18.763576 7780 raft_consensus.cc:504] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } }
I20260924 05:09:18.763712 7780 leader_election.cc:288] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [CANDIDATE]: Term 2 election: Requested vote from peers be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509), 339dac30bfc541828c4365d10e80a0ff (127.0.98.60:39493)
I20260924 05:09:18.763790 7087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "339dac30bfc541828c4365d10e80a0ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c60180f652c7462c890b08111d966b47"
I20260924 05:09:18.763839 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.763633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47720)
0924 05:09:18.763785 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.763816 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 05:09:18.763882 7087 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 339dac30bfc541828c4365d10e80a0ff in current term 2: Already voted for candidate c60180f652c7462c890b08111d966b47 in this term.
I20260924 05:09:18.763844 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c60180f652c7462c890b08111d966b47" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "339dac30bfc541828c4365d10e80a0ff"
I20260924 05:09:18.763942 7272 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c60180f652c7462c890b08111d966b47 in current term 2: Already voted for candidate 339dac30bfc541828c4365d10e80a0ff in this term.
I20260924 05:09:18.764117 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.763769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47734)
0924 05:09:18.764065 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.764095 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281}
I20260924 05:09:18.764258 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.764173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47744)
0924 05:09:18.764213 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.764236 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 05:09:18.764389 7247 leader_election.cc:334] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:18.764425 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:18.764350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47758)
0924 05:09:18.764378 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:18.764409 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 05:09:18.764458 7247 leader_election.cc:302] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 339dac30bfc541828c4365d10e80a0ff; no voters: be3fb6c102d140b789c4055d9fdd2280, c60180f652c7462c890b08111d966b47
W20260924 05:09:18.764495 7061 leader_election.cc:334] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:18.764518 7778 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 05:09:18.764535 7061 leader_election.cc:302] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c60180f652c7462c890b08111d966b47; no voters: 339dac30bfc541828c4365d10e80a0ff, be3fb6c102d140b789c4055d9fdd2280
I20260924 05:09:18.764678 7780 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 05:09:19.254935 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:19.254594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:52825)
0924 05:09:19.254831 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:19.254876 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260924 05:09:19.255213 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:19.254896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:39953)
0924 05:09:19.255129 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:19.255166 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":70,"threads_started":1}
I20260924 05:09:19.255452 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:19.255178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:54509)
0924 05:09:19.255379 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:19.255411 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":53,"threads_started":1}
I20260924 05:09:20.255635 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.255378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:60019)
0924 05:09:20.255535 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.255577 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260924 05:09:20.256081 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.255664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:45285)
0924 05:09:20.255997 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.256036 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":73,"threads_started":1}
I20260924 05:09:20.256160 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.255953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:34055)
0924 05:09:20.256084 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.256117 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":44,"threads_started":1}
I20260924 05:09:20.422852 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:20.422947 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } }
I20260924 05:09:20.423139 7794 leader_election.cc:288] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509), c60180f652c7462c890b08111d966b47 (127.0.98.61:39249)
I20260924 05:09:20.423509 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.423333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47968)
0924 05:09:20.423429 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.423467 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":61,"threads_started":1}
I20260924 05:09:20.423592 7087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "339dac30bfc541828c4365d10e80a0ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c60180f652c7462c890b08111d966b47" is_pre_election: true
I20260924 05:09:20.423692 7087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 339dac30bfc541828c4365d10e80a0ff in term 2.
I20260924 05:09:20.423928 7247 leader_election.cc:302] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339dac30bfc541828c4365d10e80a0ff, c60180f652c7462c890b08111d966b47; no voters:
I20260924 05:09:20.423987 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.423878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47980)
0924 05:09:20.423925 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.423950 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:09:20.424013 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 05:09:20.424043 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 05:09:20.424052 7247 leader_election.cc:334] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:20.424077 7794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:09:20.424767 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } }
I20260924 05:09:20.424901 7794 leader_election.cc:288] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 3 election: Requested vote from peers be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509), c60180f652c7462c890b08111d966b47 (127.0.98.61:39249)
I20260924 05:09:20.425078 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.425007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47984)
0924 05:09:20.425034 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.425057 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 05:09:20.425139 7087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "339dac30bfc541828c4365d10e80a0ff" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c60180f652c7462c890b08111d966b47"
I20260924 05:09:20.425230 7087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:09:20.425810 7087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 339dac30bfc541828c4365d10e80a0ff in term 3.
I20260924 05:09:20.425981 7247 leader_election.cc:302] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339dac30bfc541828c4365d10e80a0ff, c60180f652c7462c890b08111d966b47; no voters:
I20260924 05:09:20.426081 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 3 FOLLOWER]: Leader election won for term 3
I20260924 05:09:20.426184 7794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [term 3 LEADER]: Becoming Leader. State: Replica: 339dac30bfc541828c4365d10e80a0ff, State: Running, Role: LEADER
I20260924 05:09:20.426221 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.426119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:47990)
0924 05:09:20.426169 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.426201 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 05:09:20.426261 7794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } }
W20260924 05:09:20.426314 7247 leader_election.cc:334] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
I20260924 05:09:20.426604 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 339dac30bfc541828c4365d10e80a0ff. Latest consensus state: current_term: 3 leader_uuid: "339dac30bfc541828c4365d10e80a0ff" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:20.426700 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:20.426898 7799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:20.427333 7799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:20.427410 7799 catalog_manager.cc:1259] Loaded cluster ID: 03e5c7555e154f9095a9d3b8698904d0
I20260924 05:09:20.427444 7799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:20.427618 7799 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:20.427702 7799 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Loaded TSK: 0
I20260924 05:09:20.427838 7799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:09:20.442204 7263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:20.443833 7263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:20.444818 7087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [term 3 FOLLOWER]: Refusing update from remote peer 339dac30bfc541828c4365d10e80a0ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 05:09:20.444831 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.444728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:48002)
0924 05:09:20.444777 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.444808 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:09:20.445230 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.445085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.0.1:48016)
0924 05:09:20.445126 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.445169 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:09:20.445170 7797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 05:09:20.445314 7247 consensus_peers.cc:487] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff -> Peer be3fb6c102d140b789c4055d9fdd2280 (127.0.98.62:45509): Couldn't send request to peer be3fb6c102d140b789c4055d9fdd2280. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 05:09:20.445811 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 339dac30bfc541828c4365d10e80a0ff. Latest consensus state: current_term: 3 leader_uuid: "339dac30bfc541828c4365d10e80a0ff" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:20.445920 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:09:20.446398 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "339dac30bfc541828c4365d10e80a0ff" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:20.446468 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "339dac30bfc541828c4365d10e80a0ff" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be3fb6c102d140b789c4055d9fdd2280" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } peers { permanent_uuid: "c60180f652c7462c890b08111d966b47" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39249 } attrs { promote: false } } peers { permanent_uuid: "339dac30bfc541828c4365d10e80a0ff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39493 } attrs { promote: false } } }
I20260924 05:09:20.446593 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60180f652c7462c890b08111d966b47 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:09:20.446540 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:20.451171 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.448087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46515 (local address 127.0.0.1:39222)
0924 05:09:20.448114 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.448124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:20.448171 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:20.448613 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:20.448614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:20.448616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:20.448668 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.448670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.449334 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.449335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.450152 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.450153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.450919 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.450921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.450983 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.450989 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 05:09:20.451064 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:09:20.451109 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 05:09:20.451176 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.448353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.0.98.3:46515)
0924 05:09:20.448460 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:20.448462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:20.448470 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:20.448496 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:20.448497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:20.448508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:20.448607 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:20.448734 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.449245 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.450236 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.450835 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.450902 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.450916 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:20.451072 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:09:20.451110 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":46,"threads_started":1}
I20260924 05:09:20.451526 7649 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf95e8453df47ad962cf4f655512dc9 (DEFAULT_TABLE table=test-table [id=473e3a43c4cc417bbc0eb3b163762196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:20.451620 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.448184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34269 (local address 127.0.0.1:46520)
0924 05:09:20.448397 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.448408 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:20.448466 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:20.448657 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:20.448658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:20.448659 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:20.448704 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.448706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.449350 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.449351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.450524 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.450526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.451380 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.451381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.451452 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.451455 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:20.451497 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:09:20.451534 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260924 05:09:20.451682 7649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf95e8453df47ad962cf4f655512dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:20.451810 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.448289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.0.98.2:34269)
0924 05:09:20.448408 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:20.448410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:20.448452 (+ 42us) server_negotiation.cc:407] Connection header received
0924 05:09:20.448548 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:20.448550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:20.448561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:20.448617 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:20.448778 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.449255 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.450613 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.451206 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.451305 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.451316 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:20.451706 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 05:09:20.451746 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
I20260924 05:09:20.452092 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf95e8453df47ad962cf4f655512dc9 (DEFAULT_TABLE table=test-table [id=473e3a43c4cc417bbc0eb3b163762196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:20.452205 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf95e8453df47ad962cf4f655512dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:20.452548 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.448192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.0.98.1:43613)
0924 05:09:20.449391 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:20.449393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:20.449401 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:20.449424 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:20.449425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:20.449434 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:20.449503 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:20.449612 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.450324 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.451599 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.452212 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.452279 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.452289 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:20.452450 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 05:09:20.452484 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1178,"thread_start_us":53,"threads_started":1}
I20260924 05:09:20.452545 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.448132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.0.1:39142)
0924 05:09:20.448291 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.448300 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:20.448371 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:20.449518 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:20.449518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:20.449521 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:20.449552 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.449554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.450411 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.450412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.451520 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.451521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.452319 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.452320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.452371 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.452374 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:20.452443 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 05:09:20.452479 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20260924 05:09:20.453102 7811 tablet_bootstrap.cc:492] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Bootstrap starting.
I20260924 05:09:20.453442 7811 tablet_bootstrap.cc:654] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:20.453609 7812 tablet_bootstrap.cc:492] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: Bootstrap starting.
I20260924 05:09:20.453823 7376 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf95e8453df47ad962cf4f655512dc9 (DEFAULT_TABLE table=test-table [id=473e3a43c4cc417bbc0eb3b163762196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:20.453943 7376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf95e8453df47ad962cf4f655512dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:20.454032 7812 tablet_bootstrap.cc:654] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:20.454526 7812 tablet_bootstrap.cc:492] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: No bootstrap required, opened a new log
I20260924 05:09:20.454582 7812 ts_tablet_manager.cc:1397] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:20.454689 7812 raft_consensus.cc:348] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:20.454789 7812 raft_consensus.cc:374] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:20.454819 7812 raft_consensus.cc:729] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc364bc179174f3ab1fa7f7f1503bc2d, State: Initialized, Role: FOLLOWER
I20260924 05:09:20.454895 7812 consensus_queue.cc:260] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:20.455063 7812 ts_tablet_manager.cc:1428] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:20.455382 7811 tablet_bootstrap.cc:492] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: No bootstrap required, opened a new log
I20260924 05:09:20.455437 7811 ts_tablet_manager.cc:1397] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:20.455427 7816 tablet_bootstrap.cc:492] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e: Bootstrap starting.
I20260924 05:09:20.455487 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.455253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:54073)
0924 05:09:20.455432 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.455465 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260924 05:09:20.455544 7811 raft_consensus.cc:348] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:20.455600 7811 raft_consensus.cc:374] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:20.455623 7811 raft_consensus.cc:729] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1f6210a1d247a49dbf6a7f5ec5a5b0, State: Initialized, Role: FOLLOWER
I20260924 05:09:20.455668 7811 consensus_queue.cc:260] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:20.455817 7816 tablet_bootstrap.cc:654] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:20.455862 7811 ts_tablet_manager.cc:1428] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:20.456211 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.456118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:60903)
0924 05:09:20.456153 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.456180 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 05:09:20.456710 7816 tablet_bootstrap.cc:492] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e: No bootstrap required, opened a new log
I20260924 05:09:20.456732 7716 heartbeater.cc:503] Master 127.0.98.60:39493 was elected leader, sending a full tablet report...
I20260924 05:09:20.456791 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.456657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:51119)
0924 05:09:20.456695 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.456745 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 05:09:20.456802 7816 ts_tablet_manager.cc:1397] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:20.457051 7816 raft_consensus.cc:348] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:20.457193 7816 raft_consensus.cc:374] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:20.457237 7816 raft_consensus.cc:729] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c91da51a17b439d811e79b4f59f500e, State: Initialized, Role: FOLLOWER
I20260924 05:09:20.457315 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.457212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:43395)
0924 05:09:20.457256 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.457288 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 05:09:20.457314 7816 consensus_queue.cc:260] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:20.457499 7580 heartbeater.cc:503] Master 127.0.98.60:39493 was elected leader, sending a full tablet report...
I20260924 05:09:20.457553 7816 ts_tablet_manager.cc:1428] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:09:20.457823 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.457731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:57323)
0924 05:09:20.457766 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.457795 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 05:09:20.458220 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.458111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:43699)
0924 05:09:20.458165 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.458196 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 05:09:20.458278 7443 heartbeater.cc:503] Master 127.0.98.60:39493 was elected leader, sending a full tablet report...
I20260924 05:09:20.458454 7819 raft_consensus.cc:482] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:20.458540 7819 raft_consensus.cc:504] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:20.458859 7819 leader_election.cc:288] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1f6210a1d247a49dbf6a7f5ec5a5b0 (127.0.98.3:46515), dc364bc179174f3ab1fa7f7f1503bc2d (127.0.98.2:34269)
I20260924 05:09:20.461512 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.459012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42265 (local address 127.0.98.3:46515)
0924 05:09:20.459045 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:20.459047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:20.459234 (+ 187us) server_negotiation.cc:407] Connection header received
0924 05:09:20.459283 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:20.459284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:20.459295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:20.459347 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:20.459487 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.459920 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.460832 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.461177 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.461275 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.461284 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:20.461426 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 05:09:20.461457 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:09:20.461519 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.458959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46515 (local address 127.0.98.1:42265)
0924 05:09:20.459158 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.459168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:20.459224 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:20.459351 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:20.459352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:20.459354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:20.459406 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.459408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.460009 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.460009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.460725 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.460726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.461272 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.461273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.461333 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.461337 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:09:20.461420 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:09:20.461458 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260924 05:09:20.461566 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.459099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50099 (local address 127.0.98.2:34269)
0924 05:09:20.459133 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:20.459135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:20.459369 (+ 234us) server_negotiation.cc:407] Connection header received
0924 05:09:20.459403 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:20.459404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:20.459413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:20.459464 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:20.459555 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.459993 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.460871 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.461218 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.461310 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.461319 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:20.461475 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:09:20.461506 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:20.461612 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.459054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34269 (local address 127.0.98.1:50099)
0924 05:09:20.459305 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.459312 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:20.459356 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:20.459468 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:20.459469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:20.459470 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:20.459508 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.459510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.460074 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.460075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.460767 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:20.460769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.461314 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.461315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:20.461374 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.461378 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:09:20.461467 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 05:09:20.461506 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":74,"threads_started":1}
I20260924 05:09:20.461818 7669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf95e8453df47ad962cf4f655512dc9" candidate_uuid: "6c91da51a17b439d811e79b4f59f500e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" is_pre_election: true
I20260924 05:09:20.461902 7669 raft_consensus.cc:2454] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c91da51a17b439d811e79b4f59f500e in term 0.
I20260924 05:09:20.461938 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf95e8453df47ad962cf4f655512dc9" candidate_uuid: "6c91da51a17b439d811e79b4f59f500e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" is_pre_election: true
I20260924 05:09:20.462049 7533 raft_consensus.cc:2454] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c91da51a17b439d811e79b4f59f500e in term 0.
I20260924 05:09:20.462235 7332 leader_election.cc:302] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c91da51a17b439d811e79b4f59f500e, be1f6210a1d247a49dbf6a7f5ec5a5b0; no voters:
I20260924 05:09:20.462527 7819 raft_consensus.cc:2781] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:20.462606 7819 raft_consensus.cc:482] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:20.462649 7819 raft_consensus.cc:3037] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:20.463274 7819 raft_consensus.cc:504] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:20.463395 7819 leader_election.cc:288] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [CANDIDATE]: Term 1 election: Requested vote from peers be1f6210a1d247a49dbf6a7f5ec5a5b0 (127.0.98.3:46515), dc364bc179174f3ab1fa7f7f1503bc2d (127.0.98.2:34269)
I20260924 05:09:20.463546 7669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf95e8453df47ad962cf4f655512dc9" candidate_uuid: "6c91da51a17b439d811e79b4f59f500e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0"
I20260924 05:09:20.463580 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf95e8453df47ad962cf4f655512dc9" candidate_uuid: "6c91da51a17b439d811e79b4f59f500e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d"
I20260924 05:09:20.463636 7533 raft_consensus.cc:3037] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:20.463639 7669 raft_consensus.cc:3037] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:20.464138 7669 raft_consensus.cc:2454] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c91da51a17b439d811e79b4f59f500e in term 1.
I20260924 05:09:20.464190 7533 raft_consensus.cc:2454] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c91da51a17b439d811e79b4f59f500e in term 1.
I20260924 05:09:20.464327 7332 leader_election.cc:302] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c91da51a17b439d811e79b4f59f500e, dc364bc179174f3ab1fa7f7f1503bc2d; no voters:
I20260924 05:09:20.464507 7819 raft_consensus.cc:2781] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:20.464672 7819 raft_consensus.cc:686] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 1 LEADER]: Becoming Leader. State: Replica: 6c91da51a17b439d811e79b4f59f500e, State: Running, Role: LEADER
I20260924 05:09:20.464818 7819 consensus_queue.cc:237] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } }
I20260924 05:09:20.464973 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.464852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:33939)
0924 05:09:20.464913 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.464944 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:09:20.465432 7263 catalog_manager.cc:5495] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c91da51a17b439d811e79b4f59f500e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6c91da51a17b439d811e79b4f59f500e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c91da51a17b439d811e79b4f59f500e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 } health_report { overall_health: UNKNOWN } } }
I20260924 05:09:20.470455 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.468478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.0.98.1:43613)
0924 05:09:20.468524 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:20.468526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:20.468533 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:09:20.468602 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:20.468603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:20.468616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:20.468674 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:20.468750 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.469160 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.469824 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:20.470014 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:20.470066 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:20.470185 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:20.470272 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:20.470324 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:20.470361 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 05:09:20.470375 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:09:20.471535 7669 raft_consensus.cc:1261] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 1 FOLLOWER]: Refusing update from remote peer 6c91da51a17b439d811e79b4f59f500e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:20.471560 7533 raft_consensus.cc:1261] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [term 1 FOLLOWER]: Refusing update from remote peer 6c91da51a17b439d811e79b4f59f500e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:20.471777 7819 consensus_queue.cc:1035] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1f6210a1d247a49dbf6a7f5ec5a5b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:20.471890 7821 consensus_queue.cc:1035] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc364bc179174f3ab1fa7f7f1503bc2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:20.472517 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.472419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:59927)
0924 05:09:20.472462 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.472488 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 05:09:20.473121 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.472949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:41267)
0924 05:09:20.473062 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.473090 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 05:09:20.473845 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.473692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:42171)
0924 05:09:20.473788 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.473816 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 05:09:20.473971 7825 mvcc.cc:205] Tried to move back new op lower bound from 7332767991689773056 to 7332767991664599040. Current Snapshot: MvccSnapshot[applied={T|T < 7332767991689773056}]
I20260924 05:09:20.474483 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.474388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:42591)
0924 05:09:20.474425 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.474452 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 05:09:20.475399 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.475308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:47259)
0924 05:09:20.475340 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.475368 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 05:09:20.475895 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:20.475810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:39899)
0924 05:09:20.475847 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.475869 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:09:20.476106 7263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:20.476183 7263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:20.475906 7826 mvcc.cc:205] Tried to move back new op lower bound from 7332767991689773056 to 7332767991664599040. Current Snapshot: MvccSnapshot[applied={T|T < 7332767991689773056}]
I20260924 05:09:20.477437 7263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 339dac30bfc541828c4365d10e80a0ff: Sending DeleteTablet for 3 replicas of tablet 6cf95e8453df47ad962cf4f655512dc9
I20260924 05:09:20.477715 7376 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf95e8453df47ad962cf4f655512dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:20 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39142
I20260924 05:09:20.477733 7513 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf95e8453df47ad962cf4f655512dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:20 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46520
I20260924 05:09:20.477909 7835 tablet_replica.cc:330] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e: stopping tablet replica
I20260924 05:09:20.477967 7649 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf95e8453df47ad962cf4f655512dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:20 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39222
I20260924 05:09:20.478008 7835 raft_consensus.cc:2229] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:20.478053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 7224
I20260924 05:09:20.478091 7835 raft_consensus.cc:2258] T 6cf95e8453df47ad962cf4f655512dc9 P 6c91da51a17b439d811e79b4f59f500e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:20.479696 7837 tablet_replica.cc:330] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: stopping tablet replica
I20260924 05:09:20.479784 7837 raft_consensus.cc:2229] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:20.479832 7837 raft_consensus.cc:2258] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:20.479924 7836 tablet_replica.cc:330] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: stopping tablet replica
I20260924 05:09:20.479975 7836 raft_consensus.cc:2229] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:20.480016 7836 raft_consensus.cc:2258] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:20.480252 7837 ts_tablet_manager.cc:1905] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:20.480269 7836 ts_tablet_manager.cc:1905] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:20.482013 7837 ts_tablet_manager.cc:1918] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:20.482013 7836 ts_tablet_manager.cc:1918] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:20.482076 7836 log.cc:1196] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/6cf95e8453df47ad962cf4f655512dc9
I20260924 05:09:20.482111 7837 log.cc:1196] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/6cf95e8453df47ad962cf4f655512dc9
I20260924 05:09:20.482352 7836 ts_tablet_manager.cc:1939] T 6cf95e8453df47ad962cf4f655512dc9 P dc364bc179174f3ab1fa7f7f1503bc2d: Deleting consensus metadata
I20260924 05:09:20.482352 7837 ts_tablet_manager.cc:1939] T 6cf95e8453df47ad962cf4f655512dc9 P be1f6210a1d247a49dbf6a7f5ec5a5b0: Deleting consensus metadata
I20260924 05:09:20.482638 7250 catalog_manager.cc:4841] TS dc364bc179174f3ab1fa7f7f1503bc2d (127.0.98.2:34269): tablet 6cf95e8453df47ad962cf4f655512dc9 (table test-table [id=473e3a43c4cc417bbc0eb3b163762196]) successfully deleted
I20260924 05:09:20.482722 7250 catalog_manager.cc:4841] TS be1f6210a1d247a49dbf6a7f5ec5a5b0 (127.0.98.3:46515): tablet 6cf95e8453df47ad962cf4f655512dc9 (table test-table [id=473e3a43c4cc417bbc0eb3b163762196]) successfully deleted
I20260924 05:09:20.484867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 7450
W20260924 05:09:20.485534 7806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:20.484516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.0.1:39164)
0924 05:09:20.485454 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:20.485487 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43613: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":918}
W20260924 05:09:20.485656 7247 catalog_manager.cc:4584] TS 6c91da51a17b439d811e79b4f59f500e (127.0.98.1:43613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cf95e8453df47ad962cf4f655512dc9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43613: connect: Connection reset by peer (error 104)
I20260924 05:09:20.490711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 7586
I20260924 05:09:20.495146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 6943
I20260924 05:09:20.499876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 7131
2026-09-24T05:09:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6040 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:20 dist-test-slave-mr2f krb5kdc[7840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:20 dist-test-slave-mr2f krb5kdc[7840](info): set up 2 sockets
Sep 24 05:09:20 dist-test-slave-mr2f krb5kdc[7840](info): commencing operation
krb5kdc: starting...
W20260924 05:09:21.535187 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:21 dist-test-slave-mr2f krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 05:09:21.649616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:21.728878 7853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:21.729069 7853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:21.729090 7853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:21.730248 7853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:21.730290 7853 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 05:09:21.730304 7853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:21.730317 7853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:21.730369 7853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:21.730386 7853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:21.731889 7853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:21.732128 7853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:21.732355 7853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:21.737015 7859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:21.737015 7861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:21.737032 7853 server_base.cc:1034] running on GCE node
W20260924 05:09:21.737005 7858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:21.737445 7853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:21.737759 7853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:21.738924 7853 hybrid_clock.cc:648] HybridClock initialized: now 1790226561738912 us; error 26 us; skew 500 ppm
Sep 24 05:09:21 dist-test-slave-mr2f krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:21.748061 7853 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 05:09:21.748449 7853 webserver.cc:460] Webserver started at http://127.0.98.62:32987/ using document root <none> and password file <none>
I20260924 05:09:21.748700 7853 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:21.748752 7853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:21.748914 7853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:21.749979 7853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "09021a56a79342f08e6c68b62fe1fee6"
format_stamp: "Formatted at 2026-09-24 05:09:21 on dist-test-slave-mr2f"
server_key: "f563e63264f0eb96fca7514669c0dacb"
server_key_iv: "41fc0e3ef0cfe6610178cd8270e77b3e"
server_key_version: "encryptionkey@0"
I20260924 05:09:21.750340 7853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "09021a56a79342f08e6c68b62fe1fee6"
format_stamp: "Formatted at 2026-09-24 05:09:21 on dist-test-slave-mr2f"
server_key: "f563e63264f0eb96fca7514669c0dacb"
server_key_iv: "41fc0e3ef0cfe6610178cd8270e77b3e"
server_key_version: "encryptionkey@0"
I20260924 05:09:21.751714 7853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:21.752491 7868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:21.752676 7853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:21.752796 7853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "09021a56a79342f08e6c68b62fe1fee6"
format_stamp: "Formatted at 2026-09-24 05:09:21 on dist-test-slave-mr2f"
server_key: "f563e63264f0eb96fca7514669c0dacb"
server_key_iv: "41fc0e3ef0cfe6610178cd8270e77b3e"
server_key_version: "encryptionkey@0"
I20260924 05:09:21.752866 7853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:21.786571 7853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:21.788885 7853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:21.789000 7853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:21.793581 7853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35621
I20260924 05:09:21.793625 7930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35621 every 8 connection(s)
I20260924 05:09:21.793943 7853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:21.794500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 7853
I20260924 05:09:21.794613 7931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:21.794631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:21.794857 392 external_mini_cluster.cc:1428] Setting key df49cc184edac1bcd68d7b6c43eaf0e1
I20260924 05:09:21.797359 7931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: Bootstrap starting.
I20260924 05:09:21.797935 7931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:21.798209 7931 log.cc:824] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:21.798899 7931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: No bootstrap required, opened a new log
I20260924 05:09:21.800799 7931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09021a56a79342f08e6c68b62fe1fee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } }
I20260924 05:09:21.800959 7931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:21.801009 7931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09021a56a79342f08e6c68b62fe1fee6, State: Initialized, Role: FOLLOWER
Sep 24 05:09:21 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:21.801184 7931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09021a56a79342f08e6c68b62fe1fee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } }
I20260924 05:09:21.801280 7931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:21.801318 7931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:21.801373 7931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:21.801981 7931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09021a56a79342f08e6c68b62fe1fee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } }
I20260924 05:09:21.802089 7931 leader_election.cc:302] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09021a56a79342f08e6c68b62fe1fee6; no voters:
I20260924 05:09:21.802271 7931 leader_election.cc:288] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:21.802314 7936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:21.802480 7936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [term 1 LEADER]: Becoming Leader. State: Replica: 09021a56a79342f08e6c68b62fe1fee6, State: Running, Role: LEADER
I20260924 05:09:21.802542 7931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:21.802585 7936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09021a56a79342f08e6c68b62fe1fee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } }
I20260924 05:09:21.803026 7937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09021a56a79342f08e6c68b62fe1fee6. Latest consensus state: current_term: 1 leader_uuid: "09021a56a79342f08e6c68b62fe1fee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09021a56a79342f08e6c68b62fe1fee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } }
I20260924 05:09:21.803208 7937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:21.804210 7938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09021a56a79342f08e6c68b62fe1fee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09021a56a79342f08e6c68b62fe1fee6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } }
I20260924 05:09:21.804363 7938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6 [sys.catalog]: This master's current role is: LEADER
W20260924 05:09:21.804463 7952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:09:21.804510 7952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:09:21.804602 7945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:21.804800 7945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:21.805546 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:21.795366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.0.98.62:35621)
0924 05:09:21.795534 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:21.795549 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:21.795574 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:09:21.796063 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:21.796069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:21.796115 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:21.796272 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:21.796877 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.797812 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.798644 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.799167 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.799285 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:21.801600 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:21.801604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:21.801604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:21.801614 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:21.803678 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:21.804145 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:21.804148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:21.804148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:21.804260 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:21.804486 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:21.804487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:21.804487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:21.804808 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:21.804974 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:21.805211 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 05:09:21.805353 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260924 05:09:21.806608 7945 catalog_manager.cc:1347] Generated new cluster ID: 3e64383c125e41b78be1eaca3a86a115
I20260924 05:09:21.806660 7945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:21.822356 7945 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:21.822955 7945 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:21.826886 7945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: Generated new TSK 0
I20260924 05:09:21.827067 7945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 05:09:21.855137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35621
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:21.941252 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:21.941488 7957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:21.941529 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:21.942682 7957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:21.942745 7957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:21.942799 7957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:09:21.942843 7957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:21.944314 7957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:21.944540 7957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:21.944756 7957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:09:21.949733 7957 server_base.cc:1034] running on GCE node
W20260924 05:09:21.949766 7965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:21.949858 7963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:21.950029 7962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:21.950248 7957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:21.950451 7957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:21.951642 7957 hybrid_clock.cc:648] HybridClock initialized: now 1790226561951637 us; error 43 us; skew 500 ppm
Sep 24 05:09:21 dist-test-slave-mr2f krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:21.953994 7957 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 05:09:21.954380 7957 webserver.cc:460] Webserver started at http://127.0.98.1:36649/ using document root <none> and password file <none>
I20260924 05:09:21.954555 7957 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:21.954594 7957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:21.954679 7957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:21.955891 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "c87b7f66de404941a0ca95a890e035ad"
format_stamp: "Formatted at 2026-09-24 05:09:21 on dist-test-slave-mr2f"
server_key: "c11b102a1350371858fff53272c7e62b"
server_key_iv: "9cb8a4392bfc8570351d53cdcfcfdf5e"
server_key_version: "encryptionkey@0"
I20260924 05:09:21.956202 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c87b7f66de404941a0ca95a890e035ad"
format_stamp: "Formatted at 2026-09-24 05:09:21 on dist-test-slave-mr2f"
server_key: "c11b102a1350371858fff53272c7e62b"
server_key_iv: "9cb8a4392bfc8570351d53cdcfcfdf5e"
server_key_version: "encryptionkey@0"
I20260924 05:09:21.957592 7957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:21.958292 7972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:21.958475 7957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:21.958545 7957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "c87b7f66de404941a0ca95a890e035ad"
format_stamp: "Formatted at 2026-09-24 05:09:21 on dist-test-slave-mr2f"
server_key: "c11b102a1350371858fff53272c7e62b"
server_key_iv: "9cb8a4392bfc8570351d53cdcfcfdf5e"
server_key_version: "encryptionkey@0"
I20260924 05:09:21.958595 7957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:21.967455 7957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:21.969132 7957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:21.969247 7957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:21.969756 7957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:21.969798 7957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:21.969820 7957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:21.969832 7957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 05:09:21 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:21.977248 7957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37997
I20260924 05:09:21.977290 8087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37997 every 8 connection(s)
I20260924 05:09:21.977599 7957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:09:21.979423 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:21.970141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42573 (local address 127.0.98.62:35621)
0924 05:09:21.970209 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:21.970211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:21.970444 (+ 233us) server_negotiation.cc:407] Connection header received
0924 05:09:21.971185 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:21.971186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:21.971195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:21.971239 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:21.972148 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.972656 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.973649 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.973862 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.973959 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:21.976453 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:21.976455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:21.976456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:21.976470 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:21.978051 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:21.978459 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:21.978460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:21.978460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:21.978519 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:21.978787 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:21.978788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:21.978789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:21.978913 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:21.978989 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:21.979252 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 05:09:21.979329 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:09:21.979622 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:21.970101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:42573)
0924 05:09:21.970280 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:21.970331 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:21.971074 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:21.971327 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:21.971332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:21.971722 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:21.972025 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:21.972034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.972753 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.972754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:21.973555 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:21.973558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.973961 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.973962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:21.974083 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:21.974545 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:21.974553 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:21.976346 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:21.978144 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:21.978149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:21.978151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:21.978364 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:21.978589 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:21.978590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:21.978590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:21.978693 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:21.978980 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:21.978987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:21.979122 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 05:09:21.979230 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 05:09:21.979386 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260924 05:09:21.980146 8088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35621
I20260924 05:09:21.980247 8088 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:21.980286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 7957
I20260924 05:09:21.980401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:21.980441 8088 heartbeater.cc:511] Master 127.0.98.62:35621 requested a full tablet report, sending...
I20260924 05:09:21.980628 392 external_mini_cluster.cc:1428] Setting key eb313a00397a1d3272d5df1858edcc01
I20260924 05:09:21.981063 7884 ts_manager.cc:194] Registered new tserver with Master: c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997)
I20260924 05:09:21.981981 7884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42573
I20260924 05:09:21.987254 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:21.981136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33823 (local address 127.0.98.62:35621)
0924 05:09:21.981180 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:21.981182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:21.981316 (+ 134us) server_negotiation.cc:407] Connection header received
0924 05:09:21.981382 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:21.981383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:21.981393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:21.981449 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:21.981749 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.982316 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.983489 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.983782 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.983856 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:21.984720 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:21.984722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:21.984722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:21.984729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:21.986067 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:21.986424 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:21.986425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:21.986425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:21.986483 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:21.986702 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:21.986703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:21.986704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:21.986855 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:21.986937 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:21.987093 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:09:21.987167 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:09:21.987272 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:21.981139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:33823)
0924 05:09:21.981265 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:21.981274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:21.981327 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:21.981443 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:21.981444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:21.981615 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:21.981661 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:21.981662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.982406 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.982407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:21.983384 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:21.983387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:21.983861 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:21.983862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:21.983910 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:21.984341 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:21.984345 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:21.984614 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:21.986168 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:21.986169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:21.986170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:21.986339 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:21.986565 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:21.986566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:21.986566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:21.986623 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:21.986929 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:21.986930 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:21.986980 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 05:09:21.987067 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 05:09:21.987167 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":48,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 05:09:22.003928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35621
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:22.083705 8097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:22.083897 8097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:22.083921 8097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:22.085024 8097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:22.085065 8097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:22.085085 8097 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:09:22.085103 8097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:22.086483 8097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:22.086652 8097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:22.086886 8097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:22.091928 8103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:22.091935 8102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:22.091935 8105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:22.092442 8097 server_base.cc:1034] running on GCE node
I20260924 05:09:22.092597 8097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:22.092784 8097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:22.093920 8097 hybrid_clock.cc:648] HybridClock initialized: now 1790226562093907 us; error 29 us; skew 500 ppm
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226562, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:22.096346 8097 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 05:09:22.096760 8097 webserver.cc:460] Webserver started at http://127.0.98.2:35615/ using document root <none> and password file <none>
I20260924 05:09:22.097008 8097 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:22.097054 8097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:22.097216 8097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:22.098317 8097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "527d8f7059264350a9670377411aeaef"
format_stamp: "Formatted at 2026-09-24 05:09:22 on dist-test-slave-mr2f"
server_key: "7084c30054453c23490d949e018a9c03"
server_key_iv: "7eb781a5a92e7b00179bd5368b837dca"
server_key_version: "encryptionkey@0"
I20260924 05:09:22.098664 8097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "527d8f7059264350a9670377411aeaef"
format_stamp: "Formatted at 2026-09-24 05:09:22 on dist-test-slave-mr2f"
server_key: "7084c30054453c23490d949e018a9c03"
server_key_iv: "7eb781a5a92e7b00179bd5368b837dca"
server_key_version: "encryptionkey@0"
I20260924 05:09:22.099980 8097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:22.100687 8112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.100872 8097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:22.100963 8097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "527d8f7059264350a9670377411aeaef"
format_stamp: "Formatted at 2026-09-24 05:09:22 on dist-test-slave-mr2f"
server_key: "7084c30054453c23490d949e018a9c03"
server_key_iv: "7eb781a5a92e7b00179bd5368b837dca"
server_key_version: "encryptionkey@0"
I20260924 05:09:22.101042 8097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:22.126384 8097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:22.128257 8097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:22.128420 8097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:22.128940 8097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:22.129004 8097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.129052 8097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:22.129077 8097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226562, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:22.136191 8097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35551
I20260924 05:09:22.136559 8097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:22.136579 8227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35551 every 8 connection(s)
I20260924 05:09:22.138578 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.129457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35053 (local address 127.0.98.62:35621)
0924 05:09:22.129525 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.129528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.129736 (+ 208us) server_negotiation.cc:407] Connection header received
0924 05:09:22.130257 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.130259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.130272 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.130368 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:22.131239 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.132078 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.133100 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.133460 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.133541 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.135900 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:22.135901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:22.135901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:22.135907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:22.137503 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.137850 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.137850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.137851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.137901 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.138101 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.138102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.138102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.138194 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:22.138243 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.138459 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.138517 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:09:22.138823 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.129443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:35053)
0924 05:09:22.129605 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.129645 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.130164 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.130442 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.130446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.130834 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:22.131131 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.131138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.132188 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.132190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.132998 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.133001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.133569 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.133570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.133695 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.134267 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:22.134276 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:22.135808 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:22.137576 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.137580 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.137582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.137783 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.137959 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.137960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.137960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.138047 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.138248 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:22.138252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:22.138355 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.138433 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.138548 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 05:09:22.139009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 8097
I20260924 05:09:22.139115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:22.139307 8228 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35621
I20260924 05:09:22.139333 392 external_mini_cluster.cc:1428] Setting key 5aaee92a7e6f16096327beb42ba0b629
I20260924 05:09:22.139400 8228 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:22.139581 8228 heartbeater.cc:511] Master 127.0.98.62:35621 requested a full tablet report, sending...
I20260924 05:09:22.139938 7884 ts_manager.cc:194] Registered new tserver with Master: 527d8f7059264350a9670377411aeaef (127.0.98.2:35551)
I20260924 05:09:22.140414 7884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35053
I20260924 05:09:22.145891 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.140257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:42779)
0924 05:09:22.140449 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.140460 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.140536 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.140680 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.140681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.140856 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:22.140912 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.140913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.141541 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.141542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.142059 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.142060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.142431 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.142432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.142500 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.142922 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:22.142926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:22.143174 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:22.144918 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.144919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.144919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.145074 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.145277 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.145277 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.145277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.145323 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.145673 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:22.145674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:22.145706 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.145752 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.145821 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260924 05:09:22.145982 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.140293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42779 (local address 127.0.98.62:35621)
0924 05:09:22.140327 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.140329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.140511 (+ 182us) server_negotiation.cc:407] Connection header received
0924 05:09:22.140612 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.140613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.140622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.140675 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:22.140989 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.141447 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.142143 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.142345 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.142429 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.143259 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:22.143261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:22.143261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:22.143268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:22.144825 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.145145 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.145146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.145147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.145207 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.145391 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.145391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.145392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.145598 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:22.145673 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.145782 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.145861 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 05:09:22.165383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35621
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:22.246834 8237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:22.247021 8237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:22.247044 8237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:22.248212 8237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:22.248256 8237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:22.248277 8237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 05:09:22.248294 8237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:22.249723 8237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:22.249956 8237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:22.250180 8237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:22.255115 8243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:22.255105 8245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:22.255172 8242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:22.255424 8237 server_base.cc:1034] running on GCE node
I20260924 05:09:22.255609 8237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:22.255806 8237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:22.256971 8237 hybrid_clock.cc:648] HybridClock initialized: now 1790226562256960 us; error 28 us; skew 500 ppm
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226562, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:22.259374 8237 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 05:09:22.259816 8237 webserver.cc:460] Webserver started at http://127.0.98.3:44657/ using document root <none> and password file <none>
I20260924 05:09:22.260035 8237 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:22.260100 8237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:22.260249 8237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:22.261317 8237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "3a82fdf19149456dbd1da4b702082766"
format_stamp: "Formatted at 2026-09-24 05:09:22 on dist-test-slave-mr2f"
server_key: "602a19923b9b75597c4286ef17f8cdee"
server_key_iv: "eb6e080274b0d22776962156c4e67147"
server_key_version: "encryptionkey@0"
I20260924 05:09:22.261685 8237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3a82fdf19149456dbd1da4b702082766"
format_stamp: "Formatted at 2026-09-24 05:09:22 on dist-test-slave-mr2f"
server_key: "602a19923b9b75597c4286ef17f8cdee"
server_key_iv: "eb6e080274b0d22776962156c4e67147"
server_key_version: "encryptionkey@0"
I20260924 05:09:22.262993 8237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:09:22.263659 8252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.263837 8237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:22.263932 8237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "3a82fdf19149456dbd1da4b702082766"
format_stamp: "Formatted at 2026-09-24 05:09:22 on dist-test-slave-mr2f"
server_key: "602a19923b9b75597c4286ef17f8cdee"
server_key_iv: "eb6e080274b0d22776962156c4e67147"
server_key_version: "encryptionkey@0"
I20260924 05:09:22.264016 8237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:22.286830 8237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:22.288787 8237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:22.288954 8237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:22.289556 8237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:22.289606 8237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.289636 8237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:22.289659 8237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226562, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:22.296733 8237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35157
I20260924 05:09:22.297065 8367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35157 every 8 connection(s)
I20260924 05:09:22.297119 8237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:22.299049 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.289987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57945 (local address 127.0.98.62:35621)
0924 05:09:22.290041 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.290043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.290315 (+ 272us) server_negotiation.cc:407] Connection header received
0924 05:09:22.290887 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.290889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.290903 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.290958 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:22.291600 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.292117 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.293113 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.293308 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.293384 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.295995 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:22.295997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:22.295998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:22.296005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:22.297643 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.298078 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.298080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.298080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.298137 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.298396 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.298397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.298397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.298515 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:22.298591 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.298878 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.298954 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:09:22.299247 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.289953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:57945)
0924 05:09:22.290185 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.290213 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.290716 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.291032 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.291035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.291324 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:22.291503 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.291507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.292215 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.292216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.293009 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.293011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.293389 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.293391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.293547 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.294043 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:22.294051 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:22.295900 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:22.297739 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.297745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.297747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.297990 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.298210 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.298211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.298211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.298327 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.298583 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:22.298589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:22.298720 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.298856 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.299009 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260924 05:09:22.299743 8368 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35621
I20260924 05:09:22.299830 8368 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:22.300034 8368 heartbeater.cc:511] Master 127.0.98.62:35621 requested a full tablet report, sending...
I20260924 05:09:22.300019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 8237
I20260924 05:09:22.300132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:22.300347 392 external_mini_cluster.cc:1428] Setting key 4a0033b811b15f735668acc53dd2e7c4
I20260924 05:09:22.300673 7885 ts_manager.cc:194] Registered new tserver with Master: 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157)
I20260924 05:09:22.301093 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57945
I20260924 05:09:22.301662 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:09:22.307156 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.300808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53665 (local address 127.0.98.62:35621)
0924 05:09:22.300848 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.300850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.301362 (+ 512us) server_negotiation.cc:407] Connection header received
0924 05:09:22.301432 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.301433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.301447 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.301520 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:22.301864 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.302411 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.303195 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.303397 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.303516 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.304434 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:22.304435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:22.304436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:22.304442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:22.305991 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.306337 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.306338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.306338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.306398 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.306581 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.306582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.306582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.306717 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:22.306813 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.306997 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.307075 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:09:22.307200 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.300759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:53665)
0924 05:09:22.301304 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.301316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.301375 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.301528 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.301528 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.301716 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:22.301769 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.301771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.302521 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.302521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.303100 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.303102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.303519 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.303520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.303588 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.304048 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:22.304052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:22.304345 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:22.306099 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.306100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.306100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.306258 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.306466 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.306467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.306467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.306518 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.306837 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:22.306838 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:22.306870 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.306951 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.307057 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":63,"threads_started":1}
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 05:09:22.322808 7934 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:22.317898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.0.98.62:35621)
0924 05:09:22.317938 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.317940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.317956 (+ 16us) server_negotiation.cc:407] Connection header received
0924 05:09:22.317994 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.317995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.318003 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.318048 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:22.318563 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.319075 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.319746 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.319928 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.319976 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.322692 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50734: BlockingRecv error: recv got EOF from 127.0.0.1:50734 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:22.329406 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.323550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.0.98.62:35621)
0924 05:09:22.323625 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.323627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.323635 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:22.323693 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.323694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.323701 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.323747 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:22.324159 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.324593 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.325190 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.325363 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.325410 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.327426 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:22.327427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:22.327427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:22.327433 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:22.328518 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.328802 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.328802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.328803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.328849 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.329008 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.329009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.329009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.329096 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:22.329167 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.329244 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.329295 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:09:22.331022 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:22.332523 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:22.338682 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.334566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37997 (local address 127.0.0.1:32918)
0924 05:09:22.334739 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.334804 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.334865 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.335401 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.335401 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.335403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:22.335457 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.335459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.336723 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.336724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.337543 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.337545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.338409 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.338410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.338479 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.338489 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.338544 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.338610 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
I20260924 05:09:22.338707 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.334673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.0.98.1:37997)
0924 05:09:22.335226 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.335231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.335241 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:22.335276 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.335278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.335294 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.335533 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:22.335594 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.336633 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.337631 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.338329 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.338406 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.338476 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.338558 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.338599 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":78,"threads_started":1}
I20260924 05:09:22.339130 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.334612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35157 (local address 127.0.0.1:52576)
0924 05:09:22.334962 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.334990 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.335028 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.335255 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.335258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.335260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:22.335336 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.335337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.336633 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.336634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.337737 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.337739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.338609 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.338610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.338664 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.338667 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.338719 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.339042 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":5,"thread_start_us":74,"threads_started":1}
I20260924 05:09:22.339377 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.334878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.0.98.3:35157)
0924 05:09:22.335066 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.335071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.335081 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:22.335117 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.335119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.335137 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.335221 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:22.335423 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.336547 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.337828 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.338531 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.338597 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.338664 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.339240 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.339280 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260924 05:09:22.339439 8022 tablet_service.cc:1467] Processing CreateTablet for tablet 0e92d7eb3838460ba24d6a234fcbc196 (DEFAULT_TABLE table=test-table [id=dd0959414cf442428445e1731cfd48a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:22.339841 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e92d7eb3838460ba24d6a234fcbc196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.340036 8302 tablet_service.cc:1467] Processing CreateTablet for tablet 0e92d7eb3838460ba24d6a234fcbc196 (DEFAULT_TABLE table=test-table [id=dd0959414cf442428445e1731cfd48a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:22.340350 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e92d7eb3838460ba24d6a234fcbc196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.343106 8397 tablet_bootstrap.cc:492] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: Bootstrap starting.
I20260924 05:09:22.343612 8396 tablet_bootstrap.cc:492] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: Bootstrap starting.
I20260924 05:09:22.343683 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.334612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35551 (local address 127.0.0.1:58844)
0924 05:09:22.334834 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.334845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.334898 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.335371 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.335372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.335374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:22.335428 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.335430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.336332 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.336334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.337737 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.337739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.343419 (+ 5680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.343420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.343495 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.343499 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.343560 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.343604 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":29,"thread_start_us":92,"threads_started":1}
I20260924 05:09:22.343690 8397 tablet_bootstrap.cc:654] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.344009 8397 log.cc:824] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:22.344058 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.335054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58844 (local address 127.0.98.2:35551)
0924 05:09:22.335184 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.335188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.335198 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:22.335233 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.335234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.335248 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.335320 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:22.335554 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.336244 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.338697 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.343319 (+ 4622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.343416 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.343488 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.343594 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.343647 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260924 05:09:22.344172 8396 tablet_bootstrap.cc:654] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.344442 8396 log.cc:824] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:22.344655 8162 tablet_service.cc:1467] Processing CreateTablet for tablet 0e92d7eb3838460ba24d6a234fcbc196 (DEFAULT_TABLE table=test-table [id=dd0959414cf442428445e1731cfd48a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:22.344976 8162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e92d7eb3838460ba24d6a234fcbc196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.345194 8396 tablet_bootstrap.cc:492] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: No bootstrap required, opened a new log
I20260924 05:09:22.345273 8396 ts_tablet_manager.cc:1397] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:22.346988 8396 raft_consensus.cc:348] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.347136 8396 raft_consensus.cc:374] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.347182 8396 raft_consensus.cc:729] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87b7f66de404941a0ca95a890e035ad, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.347316 8396 consensus_queue.cc:260] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.347551 8396 ts_tablet_manager.cc:1428] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:09:22.347601 8088 heartbeater.cc:503] Master 127.0.98.62:35621 was elected leader, sending a full tablet report...
I20260924 05:09:22.348387 8401 tablet_bootstrap.cc:492] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: Bootstrap starting.
I20260924 05:09:22.348556 8397 tablet_bootstrap.cc:492] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: No bootstrap required, opened a new log
I20260924 05:09:22.348615 8397 ts_tablet_manager.cc:1397] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 05:09:22.349040 8401 tablet_bootstrap.cc:654] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.349349 8401 log.cc:824] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:22.350049 8401 tablet_bootstrap.cc:492] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: No bootstrap required, opened a new log
I20260924 05:09:22.350118 8401 ts_tablet_manager.cc:1397] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:22.350701 8397 raft_consensus.cc:348] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.350881 8397 raft_consensus.cc:374] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.350919 8397 raft_consensus.cc:729] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a82fdf19149456dbd1da4b702082766, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.351028 8397 consensus_queue.cc:260] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.351274 8397 ts_tablet_manager.cc:1428] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:09:22.351317 8368 heartbeater.cc:503] Master 127.0.98.62:35621 was elected leader, sending a full tablet report...
I20260924 05:09:22.352087 8401 raft_consensus.cc:348] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.352207 8401 raft_consensus.cc:374] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.352267 8401 raft_consensus.cc:729] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527d8f7059264350a9670377411aeaef, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.352385 8401 consensus_queue.cc:260] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.352630 8401 ts_tablet_manager.cc:1428] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:09:22.352648 8228 heartbeater.cc:503] Master 127.0.98.62:35621 was elected leader, sending a full tablet report...
W20260924 05:09:22.387367 8229 tablet.cc:2369] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:09:22.478505 8089 tablet.cc:2369] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:09:22.521437 8400 raft_consensus.cc:482] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:22.521574 8400 raft_consensus.cc:504] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.522003 8400 leader_election.cc:288] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 527d8f7059264350a9670377411aeaef (127.0.98.2:35551), 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157)
I20260924 05:09:22.525347 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.522271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36059 (local address 127.0.98.3:35157)
0924 05:09:22.522314 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.522316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.522381 (+ 65us) server_negotiation.cc:407] Connection header received
0924 05:09:22.522467 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.522468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.522482 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.522555 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:22.522814 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.523511 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.524631 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.524996 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.525051 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.525059 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.525237 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.525282 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:09:22.525355 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.522157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35157 (local address 127.0.98.1:36059)
0924 05:09:22.522298 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.522311 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.522385 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.522590 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.522591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.522594 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:22.522670 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.522671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.523599 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.523600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.524532 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.524535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.525081 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.525082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.525152 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.525156 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.525210 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.525242 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":72,"thread_start_us":81,"threads_started":1}
I20260924 05:09:22.525422 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.522235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46243 (local address 127.0.98.2:35551)
0924 05:09:22.522324 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.522326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.522432 (+ 106us) server_negotiation.cc:407] Connection header received
0924 05:09:22.522493 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.522494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.522505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.522577 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:22.522749 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.523523 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.524666 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.525019 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.525112 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.525122 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.525305 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.525347 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 05:09:22.525484 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.522169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35551 (local address 127.0.98.1:46243)
0924 05:09:22.522360 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.522369 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.522420 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.522604 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.522605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.522607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:22.522669 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.522671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.523598 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.523599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.524542 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.524546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.525133 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.525134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.525216 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.525221 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.525278 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.525324 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":75,"thread_start_us":70,"threads_started":1}
I20260924 05:09:22.525987 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e92d7eb3838460ba24d6a234fcbc196" candidate_uuid: "c87b7f66de404941a0ca95a890e035ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a82fdf19149456dbd1da4b702082766" is_pre_election: true
I20260924 05:09:22.526140 8322 raft_consensus.cc:2454] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87b7f66de404941a0ca95a890e035ad in term 0.
I20260924 05:09:22.526387 7975 leader_election.cc:302] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, c87b7f66de404941a0ca95a890e035ad; no voters:
I20260924 05:09:22.526461 8182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e92d7eb3838460ba24d6a234fcbc196" candidate_uuid: "c87b7f66de404941a0ca95a890e035ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d8f7059264350a9670377411aeaef" is_pre_election: true
I20260924 05:09:22.526516 8400 raft_consensus.cc:2781] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:22.526563 8400 raft_consensus.cc:482] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:22.526597 8400 raft_consensus.cc:3037] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.526615 8182 raft_consensus.cc:2454] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87b7f66de404941a0ca95a890e035ad in term 0.
I20260924 05:09:22.527357 8400 raft_consensus.cc:504] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.527503 8400 leader_election.cc:288] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [CANDIDATE]: Term 1 election: Requested vote from peers 527d8f7059264350a9670377411aeaef (127.0.98.2:35551), 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157)
I20260924 05:09:22.527648 8182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e92d7eb3838460ba24d6a234fcbc196" candidate_uuid: "c87b7f66de404941a0ca95a890e035ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d8f7059264350a9670377411aeaef"
I20260924 05:09:22.527660 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e92d7eb3838460ba24d6a234fcbc196" candidate_uuid: "c87b7f66de404941a0ca95a890e035ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a82fdf19149456dbd1da4b702082766"
I20260924 05:09:22.527745 8182 raft_consensus.cc:3037] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.527742 8322 raft_consensus.cc:3037] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.528463 8322 raft_consensus.cc:2454] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87b7f66de404941a0ca95a890e035ad in term 1.
I20260924 05:09:22.528488 8182 raft_consensus.cc:2454] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87b7f66de404941a0ca95a890e035ad in term 1.
I20260924 05:09:22.528635 7975 leader_election.cc:302] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, c87b7f66de404941a0ca95a890e035ad; no voters:
I20260924 05:09:22.528715 8400 raft_consensus.cc:2781] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:22.528851 8400 raft_consensus.cc:686] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 1 LEADER]: Becoming Leader. State: Replica: c87b7f66de404941a0ca95a890e035ad, State: Running, Role: LEADER
I20260924 05:09:22.528965 8400 consensus_queue.cc:237] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.529758 7885 catalog_manager.cc:5495] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad reported cstate change: term changed from 0 to 1, leader changed from <none> to c87b7f66de404941a0ca95a890e035ad (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c87b7f66de404941a0ca95a890e035ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } health_report { overall_health: UNKNOWN } } }
W20260924 05:09:22.547705 8369 tablet.cc:2369] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:09:22.564384 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.561913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.0.98.1:37997)
0924 05:09:22.561959 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.561961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.561970 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:22.562062 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.562063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.562082 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.562166 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:22.562335 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.562850 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.563614 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.563813 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.563889 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.564080 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:22.564175 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:22.564265 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.564284 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.564305 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:09:22.566030 8322 raft_consensus.cc:1261] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Refusing update from remote peer c87b7f66de404941a0ca95a890e035ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:22.566057 8182 raft_consensus.cc:1261] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Refusing update from remote peer c87b7f66de404941a0ca95a890e035ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:22.566393 8400 consensus_queue.cc:1035] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:22.566538 8407 consensus_queue.cc:1035] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:22.568770 8411 mvcc.cc:205] Tried to move back new op lower bound from 7332768000268050432 to 7332768000119300096. Current Snapshot: MvccSnapshot[applied={T|T < 7332768000268050432}]
I20260924 05:09:22.569720 8410 mvcc.cc:205] Tried to move back new op lower bound from 7332768000268050432 to 7332768000119300096. Current Snapshot: MvccSnapshot[applied={T|T < 7332768000268050432 or (T in {7332768000268050432})}]
I20260924 05:09:22.569821 8412 mvcc.cc:205] Tried to move back new op lower bound from 7332768000268050432 to 7332768000119300096. Current Snapshot: MvccSnapshot[applied={T|T < 7332768000268050432 or (T in {7332768000268050432})}]
I20260924 05:09:22.570724 7884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:22.570830 7884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:22.571548 7884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: Sending DeleteTablet for 3 replicas of tablet 0e92d7eb3838460ba24d6a234fcbc196
I20260924 05:09:22.572072 8162 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e92d7eb3838460ba24d6a234fcbc196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58844
I20260924 05:09:22.572105 7884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:22.572237 7884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:22.572335 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e92d7eb3838460ba24d6a234fcbc196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32918
I20260924 05:09:22.572518 8421 tablet_replica.cc:330] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: stopping tablet replica
I20260924 05:09:22.572508 8302 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e92d7eb3838460ba24d6a234fcbc196 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52576
I20260924 05:09:22.572664 8421 raft_consensus.cc:2229] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:22.572695 8420 tablet_replica.cc:330] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: stopping tablet replica
I20260924 05:09:22.572774 8420 raft_consensus.cc:2229] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:22.572875 8420 raft_consensus.cc:2258] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:22.572876 8421 raft_consensus.cc:2258] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:22.573210 8422 tablet_replica.cc:330] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: stopping tablet replica
I20260924 05:09:22.573318 8422 raft_consensus.cc:2229] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:22.573422 8422 raft_consensus.cc:2258] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:22.573578 8421 ts_tablet_manager.cc:1905] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:22.573668 8162 tablet_service.cc:1467] Processing CreateTablet for tablet ac09a7c64c4b4312a141897c7db5154d (DEFAULT_TABLE table=test-table [id=1eed7fc57550486fab84bdb273c4c71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:22.573688 8022 tablet_service.cc:1467] Processing CreateTablet for tablet ac09a7c64c4b4312a141897c7db5154d (DEFAULT_TABLE table=test-table [id=1eed7fc57550486fab84bdb273c4c71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:22.573777 8162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac09a7c64c4b4312a141897c7db5154d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.573801 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac09a7c64c4b4312a141897c7db5154d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.575134 8396 tablet_bootstrap.cc:492] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: Bootstrap starting.
I20260924 05:09:22.575157 8401 tablet_bootstrap.cc:492] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: Bootstrap starting.
I20260924 05:09:22.575505 8422 ts_tablet_manager.cc:1905] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:22.575552 8396 tablet_bootstrap.cc:654] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.575574 8401 tablet_bootstrap.cc:654] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.576114 8302 tablet_service.cc:1467] Processing CreateTablet for tablet ac09a7c64c4b4312a141897c7db5154d (DEFAULT_TABLE table=test-table [id=1eed7fc57550486fab84bdb273c4c71a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:22.576221 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac09a7c64c4b4312a141897c7db5154d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.576220 8421 ts_tablet_manager.cc:1918] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:22.576453 8421 log.cc:1196] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/0e92d7eb3838460ba24d6a234fcbc196
I20260924 05:09:22.576759 8421 ts_tablet_manager.cc:1939] T 0e92d7eb3838460ba24d6a234fcbc196 P c87b7f66de404941a0ca95a890e035ad: Deleting consensus metadata
I20260924 05:09:22.577200 7872 catalog_manager.cc:4841] TS c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997): tablet 0e92d7eb3838460ba24d6a234fcbc196 (table test-table [id=dd0959414cf442428445e1731cfd48a3]) successfully deleted
I20260924 05:09:22.577229 8422 ts_tablet_manager.cc:1918] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:22.577327 8422 log.cc:1196] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/0e92d7eb3838460ba24d6a234fcbc196
I20260924 05:09:22.577418 8397 tablet_bootstrap.cc:492] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: Bootstrap starting.
I20260924 05:09:22.577745 8401 tablet_bootstrap.cc:492] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: No bootstrap required, opened a new log
I20260924 05:09:22.577795 8397 tablet_bootstrap.cc:654] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.577807 8401 ts_tablet_manager.cc:1397] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 05:09:22.577963 8401 raft_consensus.cc:348] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.578045 8401 raft_consensus.cc:374] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.578047 8396 tablet_bootstrap.cc:492] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: No bootstrap required, opened a new log
I20260924 05:09:22.578074 8401 raft_consensus.cc:729] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527d8f7059264350a9670377411aeaef, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.578088 8396 ts_tablet_manager.cc:1397] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 05:09:22.578121 8401 consensus_queue.cc:260] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.578223 8401 ts_tablet_manager.cc:1428] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.578217 8396 raft_consensus.cc:348] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.578253 8422 ts_tablet_manager.cc:1939] T 0e92d7eb3838460ba24d6a234fcbc196 P 3a82fdf19149456dbd1da4b702082766: Deleting consensus metadata
I20260924 05:09:22.578276 8396 raft_consensus.cc:374] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.578298 8396 raft_consensus.cc:729] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87b7f66de404941a0ca95a890e035ad, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.578343 8396 consensus_queue.cc:260] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.578418 8396 ts_tablet_manager.cc:1428] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.578652 7871 catalog_manager.cc:4841] TS 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157): tablet 0e92d7eb3838460ba24d6a234fcbc196 (table test-table [id=dd0959414cf442428445e1731cfd48a3]) successfully deleted
I20260924 05:09:22.579105 8397 tablet_bootstrap.cc:492] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: No bootstrap required, opened a new log
I20260924 05:09:22.579124 8420 ts_tablet_manager.cc:1905] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:22.579151 8397 ts_tablet_manager.cc:1397] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:22.579290 8397 raft_consensus.cc:348] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.579351 8397 raft_consensus.cc:374] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.579368 8397 raft_consensus.cc:729] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a82fdf19149456dbd1da4b702082766, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.579433 8397 consensus_queue.cc:260] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.579524 8397 ts_tablet_manager.cc:1428] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.580585 8420 ts_tablet_manager.cc:1918] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:22.580659 8420 log.cc:1196] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/0e92d7eb3838460ba24d6a234fcbc196
I20260924 05:09:22.580893 8420 ts_tablet_manager.cc:1939] T 0e92d7eb3838460ba24d6a234fcbc196 P 527d8f7059264350a9670377411aeaef: Deleting consensus metadata
I20260924 05:09:22.581285 7870 catalog_manager.cc:4841] TS 527d8f7059264350a9670377411aeaef (127.0.98.2:35551): tablet 0e92d7eb3838460ba24d6a234fcbc196 (table test-table [id=dd0959414cf442428445e1731cfd48a3]) successfully deleted
I20260924 05:09:22.586102 8407 raft_consensus.cc:482] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:22.586177 8407 raft_consensus.cc:504] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.586328 8407 leader_election.cc:288] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 527d8f7059264350a9670377411aeaef (127.0.98.2:35551), 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157)
I20260924 05:09:22.586449 8182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac09a7c64c4b4312a141897c7db5154d" candidate_uuid: "c87b7f66de404941a0ca95a890e035ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d8f7059264350a9670377411aeaef" is_pre_election: true
I20260924 05:09:22.586465 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac09a7c64c4b4312a141897c7db5154d" candidate_uuid: "c87b7f66de404941a0ca95a890e035ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a82fdf19149456dbd1da4b702082766" is_pre_election: true
I20260924 05:09:22.586524 8182 raft_consensus.cc:2454] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87b7f66de404941a0ca95a890e035ad in term 0.
I20260924 05:09:22.586535 8322 raft_consensus.cc:2454] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87b7f66de404941a0ca95a890e035ad in term 0.
I20260924 05:09:22.586658 7975 leader_election.cc:302] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, c87b7f66de404941a0ca95a890e035ad; no voters:
I20260924 05:09:22.586788 8407 raft_consensus.cc:2781] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:22.586858 8407 raft_consensus.cc:482] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:22.586892 8407 raft_consensus.cc:3037] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.587479 8407 raft_consensus.cc:504] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.587623 8407 leader_election.cc:288] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [CANDIDATE]: Term 1 election: Requested vote from peers 527d8f7059264350a9670377411aeaef (127.0.98.2:35551), 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157)
I20260924 05:09:22.587725 8182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac09a7c64c4b4312a141897c7db5154d" candidate_uuid: "c87b7f66de404941a0ca95a890e035ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d8f7059264350a9670377411aeaef"
I20260924 05:09:22.587810 8182 raft_consensus.cc:3037] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.587800 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac09a7c64c4b4312a141897c7db5154d" candidate_uuid: "c87b7f66de404941a0ca95a890e035ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a82fdf19149456dbd1da4b702082766"
I20260924 05:09:22.587862 8322 raft_consensus.cc:3037] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.588376 8322 raft_consensus.cc:2454] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87b7f66de404941a0ca95a890e035ad in term 1.
I20260924 05:09:22.588434 8182 raft_consensus.cc:2454] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87b7f66de404941a0ca95a890e035ad in term 1.
I20260924 05:09:22.588549 7975 leader_election.cc:302] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, c87b7f66de404941a0ca95a890e035ad; no voters:
I20260924 05:09:22.588639 8407 raft_consensus.cc:2781] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:22.588680 8407 raft_consensus.cc:686] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 1 LEADER]: Becoming Leader. State: Replica: c87b7f66de404941a0ca95a890e035ad, State: Running, Role: LEADER
I20260924 05:09:22.588727 8407 consensus_queue.cc:237] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.589147 7885 catalog_manager.cc:5495] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad reported cstate change: term changed from 0 to 1, leader changed from <none> to c87b7f66de404941a0ca95a890e035ad (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c87b7f66de404941a0ca95a890e035ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } health_report { overall_health: UNKNOWN } } }
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:22.622828 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.615580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.0.98.62:35621)
0924 05:09:22.615674 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.615676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.615826 (+ 150us) server_negotiation.cc:407] Connection header received
0924 05:09:22.615916 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.615916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.615926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.615980 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:22.616422 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.616910 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.617596 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.617777 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.617850 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.620251 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:22.620253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:22.620253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:22.620259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:22.621564 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.621956 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.621957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.621957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.622010 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.622248 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.622249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.622249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.622358 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:22.622433 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.622583 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.622667 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 05:09:22.622920 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.615524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:50748)
0924 05:09:22.615747 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.615756 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.615845 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.615988 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.615989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.616299 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:22.616343 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.616345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.616996 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.616996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.617511 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.617512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.617847 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.617847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.617907 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.618473 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:22.618478 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:22.620130 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:22.621661 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.621667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.621669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.621864 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.622117 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:22.622118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:22.622118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:22.622184 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:22.622424 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:22.622430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:22.622475 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.622563 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.622746 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":78,"thread_start_us":120,"threads_started":1}
I20260924 05:09:22.625052 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50748:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 05:09:22.625267 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:22.626717 8022 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3caa2e07f14126b73f452e9d1d97cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff36556010eb45eca600c214370d5cf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:09:22.626817 8302 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3caa2e07f14126b73f452e9d1d97cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff36556010eb45eca600c214370d5cf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:09:22.626911 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3caa2e07f14126b73f452e9d1d97cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.626962 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3caa2e07f14126b73f452e9d1d97cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.628115 8162 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3caa2e07f14126b73f452e9d1d97cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff36556010eb45eca600c214370d5cf8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 05:09:22.628237 8162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3caa2e07f14126b73f452e9d1d97cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.628376 8397 tablet_bootstrap.cc:492] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: Bootstrap starting.
I20260924 05:09:22.628381 8396 tablet_bootstrap.cc:492] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: Bootstrap starting.
I20260924 05:09:22.628762 8396 tablet_bootstrap.cc:654] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.628779 8397 tablet_bootstrap.cc:654] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.629601 8397 tablet_bootstrap.cc:492] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: No bootstrap required, opened a new log
I20260924 05:09:22.629657 8397 ts_tablet_manager.cc:1397] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:22.629793 8401 tablet_bootstrap.cc:492] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: Bootstrap starting.
I20260924 05:09:22.629840 8397 raft_consensus.cc:348] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.629925 8397 raft_consensus.cc:374] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.629951 8397 raft_consensus.cc:729] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a82fdf19149456dbd1da4b702082766, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.629999 8397 consensus_queue.cc:260] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.630107 8397 ts_tablet_manager.cc:1428] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:22.630179 8401 tablet_bootstrap.cc:654] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.630250 8403 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.630340 8403 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:09:22.630357 8396 tablet_bootstrap.cc:492] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: No bootstrap required, opened a new log
I20260924 05:09:22.630394 8396 ts_tablet_manager.cc:1397] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:22.630551 8396 raft_consensus.cc:348] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.630625 8396 raft_consensus.cc:374] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.630646 8396 raft_consensus.cc:729] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87b7f66de404941a0ca95a890e035ad, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.630681 8396 consensus_queue.cc:260] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.630764 8396 ts_tablet_manager.cc:1428] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.630817 8407 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.630891 8407 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:09:22.631551 8401 tablet_bootstrap.cc:492] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: No bootstrap required, opened a new log
I20260924 05:09:22.631611 8401 ts_tablet_manager.cc:1397] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:22.631747 8401 raft_consensus.cc:348] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.631812 8401 raft_consensus.cc:374] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.631830 8401 raft_consensus.cc:729] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527d8f7059264350a9670377411aeaef, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.631888 8401 consensus_queue.cc:260] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.631983 8401 ts_tablet_manager.cc:1428] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.632052 8404 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.632153 8404 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:09:22.741408 8404 raft_consensus.cc:482] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:22.741524 8404 raft_consensus.cc:504] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.741884 8404 leader_election.cc:288] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997), 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157)
I20260924 05:09:22.744854 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.742126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39329 (local address 127.0.98.3:35157)
0924 05:09:22.742172 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.742175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.742218 (+ 43us) server_negotiation.cc:407] Connection header received
0924 05:09:22.742304 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.742305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.742316 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.742383 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:22.742551 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.743073 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.744015 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.744513 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.744580 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.744589 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.744752 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.744783 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:09:22.744854 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.742132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42851 (local address 127.0.98.1:37997)
0924 05:09:22.742188 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.742190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.742260 (+ 70us) server_negotiation.cc:407] Connection header received
0924 05:09:22.742313 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.742314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.742322 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.742373 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:22.742550 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.743081 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.744033 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.744517 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.744581 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.744590 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.744757 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.744788 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:09:22.744874 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.742045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35157 (local address 127.0.98.2:39329)
0924 05:09:22.742166 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.742177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.742234 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.742386 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.742387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.742389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:22.742450 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.742452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.743148 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.743149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.743922 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.743924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.744589 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.744589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.744655 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.744685 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.744746 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.744785 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260924 05:09:22.744879 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.742048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37997 (local address 127.0.98.2:42851)
0924 05:09:22.742182 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.742192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.742257 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.742375 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.742375 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.742377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:22.742462 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.742464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.743161 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.743162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.743933 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.743935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.744591 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.744592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.744656 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.744678 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.744751 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.744794 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":65,"thread_start_us":71,"threads_started":1}
I20260924 05:09:22.745074 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3caa2e07f14126b73f452e9d1d97cb" candidate_uuid: "527d8f7059264350a9670377411aeaef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a82fdf19149456dbd1da4b702082766" is_pre_election: true
I20260924 05:09:22.745144 8322 raft_consensus.cc:2454] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527d8f7059264350a9670377411aeaef in term 0.
I20260924 05:09:22.745319 8115 leader_election.cc:302] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, 527d8f7059264350a9670377411aeaef; no voters:
I20260924 05:09:22.745332 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3caa2e07f14126b73f452e9d1d97cb" candidate_uuid: "527d8f7059264350a9670377411aeaef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87b7f66de404941a0ca95a890e035ad" is_pre_election: true
I20260924 05:09:22.745427 8404 raft_consensus.cc:2781] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:22.745466 8404 raft_consensus.cc:482] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:22.745471 8042 raft_consensus.cc:2454] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527d8f7059264350a9670377411aeaef in term 0.
I20260924 05:09:22.745494 8404 raft_consensus.cc:3037] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.746122 8404 raft_consensus.cc:504] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.746294 8404 leader_election.cc:288] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [CANDIDATE]: Term 1 election: Requested vote from peers c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997), 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157)
I20260924 05:09:22.746415 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3caa2e07f14126b73f452e9d1d97cb" candidate_uuid: "527d8f7059264350a9670377411aeaef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87b7f66de404941a0ca95a890e035ad"
I20260924 05:09:22.746428 8322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3caa2e07f14126b73f452e9d1d97cb" candidate_uuid: "527d8f7059264350a9670377411aeaef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a82fdf19149456dbd1da4b702082766"
I20260924 05:09:22.746491 8322 raft_consensus.cc:3037] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.746506 8042 raft_consensus.cc:3037] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.747105 8322 raft_consensus.cc:2454] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d8f7059264350a9670377411aeaef in term 1.
I20260924 05:09:22.747251 8115 leader_election.cc:302] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, 527d8f7059264350a9670377411aeaef; no voters:
I20260924 05:09:22.747339 8042 raft_consensus.cc:2454] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527d8f7059264350a9670377411aeaef in term 1.
I20260924 05:09:22.747352 8404 raft_consensus.cc:2781] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:22.747457 8404 raft_consensus.cc:686] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [term 1 LEADER]: Becoming Leader. State: Replica: 527d8f7059264350a9670377411aeaef, State: Running, Role: LEADER
I20260924 05:09:22.747520 8404 consensus_queue.cc:237] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.747812 8438 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: TxnStatusTablet state changed. Reason: New leader 527d8f7059264350a9670377411aeaef. Latest consensus state: current_term: 1 leader_uuid: "527d8f7059264350a9670377411aeaef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.747913 8438 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:09:22.748085 8440 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 05:09:22.748202 8440 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 05:09:22.748162 7885 catalog_manager.cc:5495] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef reported cstate change: term changed from 0 to 1, leader changed from <none> to 527d8f7059264350a9670377411aeaef (127.0.98.2). New cstate: current_term: 1 leader_uuid: "527d8f7059264350a9670377411aeaef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } health_report { overall_health: UNKNOWN } } }
I20260924 05:09:22.777912 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.775397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35551 (local address 127.0.0.1:58850)
0924 05:09:22.775468 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.775479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.775524 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.775696 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.775696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.775698 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:09:22.775759 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.775761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.776418 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.776419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.776990 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.776992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.777374 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.777375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.777442 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.777462 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:09:22.777741 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:09:22.777749 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.777789 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.777838 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:09:22.777912 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.775484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.0.98.2:35551)
0924 05:09:22.775532 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.775534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.775543 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:22.775610 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.775611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.775628 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.775691 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:22.775837 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.776328 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.777086 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.777291 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.777374 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.777564 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:22.777668 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:22.777742 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.777811 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.777832 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:09:22.779129 8322 raft_consensus.cc:1261] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Refusing update from remote peer 527d8f7059264350a9670377411aeaef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:22.779263 8042 raft_consensus.cc:1261] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Refusing update from remote peer 527d8f7059264350a9670377411aeaef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:22.779373 8438 consensus_queue.cc:1035] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:22.779531 8404 consensus_queue.cc:1035] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:22.780144 8403 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: TxnStatusTablet state changed. Reason: New leader 527d8f7059264350a9670377411aeaef. Latest consensus state: current_term: 1 leader_uuid: "527d8f7059264350a9670377411aeaef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.780251 8403 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:09:22.780249 8407 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: TxnStatusTablet state changed. Reason: New leader 527d8f7059264350a9670377411aeaef. Latest consensus state: current_term: 1 leader_uuid: "527d8f7059264350a9670377411aeaef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.780354 8407 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:09:22.780781 8404 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "527d8f7059264350a9670377411aeaef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.780826 8403 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "527d8f7059264350a9670377411aeaef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.780870 8404 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:09:22.780895 8403 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P 3a82fdf19149456dbd1da4b702082766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:09:22.781888 8438 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "527d8f7059264350a9670377411aeaef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.782032 8438 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P 527d8f7059264350a9670377411aeaef: This TxnStatusTablet replica's current role is: LEADER
I20260924 05:09:22.782255 8407 tablet_replica.cc:439] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "527d8f7059264350a9670377411aeaef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } } }
I20260924 05:09:22.782356 8407 tablet_replica.cc:442] T 6c3caa2e07f14126b73f452e9d1d97cb P c87b7f66de404941a0ca95a890e035ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 05:09:22.786960 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.784424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58321 (local address 127.0.98.2:35551)
0924 05:09:22.784480 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.784482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.784575 (+ 93us) server_negotiation.cc:407] Connection header received
0924 05:09:22.784646 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.784647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.784653 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.784692 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:22.784812 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.785234 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.786327 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.786494 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.786544 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.786715 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:22.786738 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:22.786843 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.786903 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.786917 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:09:22.787182 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.784350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35551 (local address 127.0.98.1:58321)
0924 05:09:22.784502 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.784513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.784591 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.784704 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.784705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.784707 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:09:22.784753 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.784755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.785303 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.785304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.786244 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.786247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.786565 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.786566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.786623 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.786638 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:09:22.786848 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:09:22.786849 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.786902 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.786957 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260924 05:09:22.789079 8322 raft_consensus.cc:1261] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Refusing update from remote peer c87b7f66de404941a0ca95a890e035ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:22.789101 8182 raft_consensus.cc:1261] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Refusing update from remote peer c87b7f66de404941a0ca95a890e035ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:22.789276 8400 consensus_queue.cc:1035] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:22.789355 8407 consensus_queue.cc:1035] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:22.798594 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.796096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46225 (local address 127.0.98.1:37997)
0924 05:09:22.796169 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.796171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.796177 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:09:22.796195 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.796196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.796205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.796286 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:22.796451 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.796894 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.797656 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.798045 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.798101 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.798350 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:22.798383 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:22.798447 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.798515 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.798534 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 05:09:22.798625 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.795916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37997 (local address 127.0.98.2:46225)
0924 05:09:22.796046 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.796056 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.796111 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.796290 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.796290 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.796292 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:09:22.796365 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.796367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.796973 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.796974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.797561 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.797562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.798200 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.798201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.798261 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.798275 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:09:22.798452 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:09:22.798453 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.798509 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.798547 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
I20260924 05:09:22.808024 7884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:22.808102 7884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:22.808662 7884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: Sending DeleteTablet for 3 replicas of tablet ac09a7c64c4b4312a141897c7db5154d
I20260924 05:09:22.808884 8162 tablet_service.cc:1514] Processing DeleteTablet for tablet ac09a7c64c4b4312a141897c7db5154d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58844
I20260924 05:09:22.808903 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet ac09a7c64c4b4312a141897c7db5154d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32918
I20260924 05:09:22.808979 8421 tablet_replica.cc:330] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: stopping tablet replica
I20260924 05:09:22.808981 8420 tablet_replica.cc:330] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: stopping tablet replica
I20260924 05:09:22.809028 8421 raft_consensus.cc:2229] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:22.809036 8420 raft_consensus.cc:2229] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:22.809070 8420 raft_consensus.cc:2258] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:22.809089 8421 raft_consensus.cc:2258] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:22.809409 8421 ts_tablet_manager.cc:1905] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:22.809477 8302 tablet_service.cc:1514] Processing DeleteTablet for tablet ac09a7c64c4b4312a141897c7db5154d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52576
I20260924 05:09:22.809597 8420 ts_tablet_manager.cc:1905] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:22.811095 8421 ts_tablet_manager.cc:1918] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:09:22.811156 8421 log.cc:1196] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/ac09a7c64c4b4312a141897c7db5154d
I20260924 05:09:22.811159 8420 ts_tablet_manager.cc:1918] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:09:22.811203 8420 log.cc:1196] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/ac09a7c64c4b4312a141897c7db5154d
I20260924 05:09:22.811378 8421 ts_tablet_manager.cc:1939] T ac09a7c64c4b4312a141897c7db5154d P c87b7f66de404941a0ca95a890e035ad: Deleting consensus metadata
I20260924 05:09:22.811390 8420 ts_tablet_manager.cc:1939] T ac09a7c64c4b4312a141897c7db5154d P 527d8f7059264350a9670377411aeaef: Deleting consensus metadata
I20260924 05:09:22.811497 8422 tablet_replica.cc:330] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: stopping tablet replica
I20260924 05:09:22.811546 8422 raft_consensus.cc:2229] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:22.811576 8422 raft_consensus.cc:2258] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:22.811789 7870 catalog_manager.cc:4841] TS 527d8f7059264350a9670377411aeaef (127.0.98.2:35551): tablet ac09a7c64c4b4312a141897c7db5154d (table test-table [id=1eed7fc57550486fab84bdb273c4c71a]) successfully deleted
I20260924 05:09:22.811822 7872 catalog_manager.cc:4841] TS c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997): tablet ac09a7c64c4b4312a141897c7db5154d (table test-table [id=1eed7fc57550486fab84bdb273c4c71a]) successfully deleted
I20260924 05:09:22.812043 8422 ts_tablet_manager.cc:1905] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:22.813124 8422 ts_tablet_manager.cc:1918] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:09:22.813169 8422 log.cc:1196] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/ac09a7c64c4b4312a141897c7db5154d
I20260924 05:09:22.813315 8422 ts_tablet_manager.cc:1939] T ac09a7c64c4b4312a141897c7db5154d P 3a82fdf19149456dbd1da4b702082766: Deleting consensus metadata
I20260924 05:09:22.813760 7871 catalog_manager.cc:4841] TS 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157): tablet ac09a7c64c4b4312a141897c7db5154d (table test-table [id=1eed7fc57550486fab84bdb273c4c71a]) successfully deleted
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 05:09:22.835043 7934 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:22.829052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.0.98.62:35621)
0924 05:09:22.829120 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.829122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.829238 (+ 116us) server_negotiation.cc:407] Connection header received
0924 05:09:22.829327 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.829328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.829336 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.829385 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:22.830091 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.830593 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.831703 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.831882 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.831934 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.834980 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50750: BlockingRecv error: recv got EOF from 127.0.0.1:50750 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 05:09:22 dist-test-slave-mr2f krb5kdc[7840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 05:09:22.843088 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.835773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.0.98.62:35621)
0924 05:09:22.835813 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.835814 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.835951 (+ 137us) server_negotiation.cc:407] Connection header received
0924 05:09:22.835974 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.835975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.835985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.836040 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:22.836479 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.837199 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.837845 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.838170 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.838246 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.840425 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:22.840427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:22.840427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:22.840434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:22.841936 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.842267 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.842268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.842269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.842330 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:22.842501 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:22.842501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:22.842502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:22.842678 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:22.842801 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.842919 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.843001 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:09:22.844538 7884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:22.844705 7884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:22.845965 8022 tablet_service.cc:1467] Processing CreateTablet for tablet b2310d8713a34027b9f18896bcedb2ae (DEFAULT_TABLE table=test-table [id=ed7481be3e744d238bc1ca2372768bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:22.846019 8302 tablet_service.cc:1467] Processing CreateTablet for tablet b2310d8713a34027b9f18896bcedb2ae (DEFAULT_TABLE table=test-table [id=ed7481be3e744d238bc1ca2372768bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:22.846045 8162 tablet_service.cc:1467] Processing CreateTablet for tablet b2310d8713a34027b9f18896bcedb2ae (DEFAULT_TABLE table=test-table [id=ed7481be3e744d238bc1ca2372768bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:22.846102 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2310d8713a34027b9f18896bcedb2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.846170 8162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2310d8713a34027b9f18896bcedb2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.846174 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2310d8713a34027b9f18896bcedb2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:22.847352 8397 tablet_bootstrap.cc:492] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: Bootstrap starting.
I20260924 05:09:22.847472 8396 tablet_bootstrap.cc:492] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: Bootstrap starting.
I20260924 05:09:22.847757 8397 tablet_bootstrap.cc:654] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.847810 8396 tablet_bootstrap.cc:654] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.848083 8401 tablet_bootstrap.cc:492] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: Bootstrap starting.
I20260924 05:09:22.848344 8396 tablet_bootstrap.cc:492] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: No bootstrap required, opened a new log
I20260924 05:09:22.848407 8396 ts_tablet_manager.cc:1397] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:22.848479 8401 tablet_bootstrap.cc:654] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:22.848519 8396 raft_consensus.cc:348] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.848592 8396 raft_consensus.cc:374] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.848618 8396 raft_consensus.cc:729] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87b7f66de404941a0ca95a890e035ad, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.848656 8396 consensus_queue.cc:260] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.848745 8396 ts_tablet_manager.cc:1428] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:22.849188 8397 tablet_bootstrap.cc:492] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: No bootstrap required, opened a new log
I20260924 05:09:22.849237 8397 ts_tablet_manager.cc:1397] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:22.849277 8401 tablet_bootstrap.cc:492] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: No bootstrap required, opened a new log
I20260924 05:09:22.849339 8401 ts_tablet_manager.cc:1397] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:22.849383 8397 raft_consensus.cc:348] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.849465 8397 raft_consensus.cc:374] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.849488 8397 raft_consensus.cc:729] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a82fdf19149456dbd1da4b702082766, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.849493 8401 raft_consensus.cc:348] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.849529 8397 consensus_queue.cc:260] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.849560 8401 raft_consensus.cc:374] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:22.849586 8401 raft_consensus.cc:729] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527d8f7059264350a9670377411aeaef, State: Initialized, Role: FOLLOWER
I20260924 05:09:22.849609 8397 ts_tablet_manager.cc:1428] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:22.849632 8401 consensus_queue.cc:260] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.849720 8401 ts_tablet_manager.cc:1428] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:22.977712 8403 raft_consensus.cc:482] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:22.977819 8403 raft_consensus.cc:504] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.978149 8403 leader_election.cc:288] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997), 527d8f7059264350a9670377411aeaef (127.0.98.2:35551)
I20260924 05:09:22.982124 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.978363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34721 (local address 127.0.98.1:37997)
0924 05:09:22.978412 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.978414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.978503 (+ 89us) server_negotiation.cc:407] Connection header received
0924 05:09:22.978646 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.978647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.978662 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.978734 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:22.978917 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.979572 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.981228 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.981770 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.981849 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.981863 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.982031 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.982067 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:09:22.982124 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.978495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56191 (local address 127.0.98.2:35551)
0924 05:09:22.978542 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.978544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.978620 (+ 76us) server_negotiation.cc:407] Connection header received
0924 05:09:22.978715 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.978716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.978725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.978800 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:22.979001 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.979730 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.981229 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.981819 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.981883 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.981891 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.982031 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.982062 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:09:22.982170 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.978309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37997 (local address 127.0.98.3:34721)
0924 05:09:22.978437 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.978448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.978532 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.978739 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.978740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.978742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:22.978844 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.978846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.979654 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.979654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.981138 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.981141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.981883 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.981884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.981955 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.981959 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.982017 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.982065 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260924 05:09:22.982178 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.978319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35551 (local address 127.0.98.3:56191)
0924 05:09:22.978558 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:22.978568 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:22.978645 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:22.978812 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:22.978813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:22.978815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:22.978920 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.978922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.979835 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.979836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.981138 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:22.981141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.981894 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.981895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:22.981964 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.981966 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:09:22.982017 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:09:22.982065 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":11,"thread_start_us":59,"threads_started":1}
I20260924 05:09:22.982436 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2310d8713a34027b9f18896bcedb2ae" candidate_uuid: "3a82fdf19149456dbd1da4b702082766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87b7f66de404941a0ca95a890e035ad" is_pre_election: true
I20260924 05:09:22.982522 8042 raft_consensus.cc:2454] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a82fdf19149456dbd1da4b702082766 in term 0.
I20260924 05:09:22.982586 8182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2310d8713a34027b9f18896bcedb2ae" candidate_uuid: "3a82fdf19149456dbd1da4b702082766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d8f7059264350a9670377411aeaef" is_pre_election: true
I20260924 05:09:22.982667 8182 raft_consensus.cc:2454] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a82fdf19149456dbd1da4b702082766 in term 0.
I20260924 05:09:22.982676 8256 leader_election.cc:302] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, c87b7f66de404941a0ca95a890e035ad; no voters:
I20260924 05:09:22.982812 8403 raft_consensus.cc:2781] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:22.982848 8403 raft_consensus.cc:482] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:22.982882 8403 raft_consensus.cc:3037] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.983515 8403 raft_consensus.cc:504] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.983645 8403 leader_election.cc:288] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [CANDIDATE]: Term 1 election: Requested vote from peers c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997), 527d8f7059264350a9670377411aeaef (127.0.98.2:35551)
I20260924 05:09:22.983796 8182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2310d8713a34027b9f18896bcedb2ae" candidate_uuid: "3a82fdf19149456dbd1da4b702082766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d8f7059264350a9670377411aeaef"
I20260924 05:09:22.983799 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2310d8713a34027b9f18896bcedb2ae" candidate_uuid: "3a82fdf19149456dbd1da4b702082766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87b7f66de404941a0ca95a890e035ad"
I20260924 05:09:22.983871 8042 raft_consensus.cc:3037] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.983875 8182 raft_consensus.cc:3037] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:22.984447 8042 raft_consensus.cc:2454] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a82fdf19149456dbd1da4b702082766 in term 1.
I20260924 05:09:22.984457 8182 raft_consensus.cc:2454] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a82fdf19149456dbd1da4b702082766 in term 1.
I20260924 05:09:22.984606 8256 leader_election.cc:302] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, 527d8f7059264350a9670377411aeaef, c87b7f66de404941a0ca95a890e035ad; no voters:
I20260924 05:09:22.984704 8403 raft_consensus.cc:2781] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:22.984827 8403 raft_consensus.cc:686] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 1 LEADER]: Becoming Leader. State: Replica: 3a82fdf19149456dbd1da4b702082766, State: Running, Role: LEADER
I20260924 05:09:22.984910 8403 consensus_queue.cc:237] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:22.985555 7884 catalog_manager.cc:5495] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a82fdf19149456dbd1da4b702082766 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3a82fdf19149456dbd1da4b702082766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } health_report { overall_health: HEALTHY } } }
I20260924 05:09:22.993992 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:22.991938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.0.98.3:35157)
0924 05:09:22.992003 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:22.992005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:22.992013 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:22.992043 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:22.992043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:22.992065 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:22.992122 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:22.992226 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.992677 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.993348 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:22.993567 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:22.993631 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:22.993756 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:22.993843 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:22.993896 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:22.993930 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 05:09:22.993944 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 05:09:22.994848 8182 raft_consensus.cc:1261] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Refusing update from remote peer 3a82fdf19149456dbd1da4b702082766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:22.994925 8042 raft_consensus.cc:1261] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Refusing update from remote peer 3a82fdf19149456dbd1da4b702082766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:22.995100 8403 consensus_queue.cc:1035] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:22.995255 8472 consensus_queue.cc:1035] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:22.998271 7884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:22.998368 7884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:22.998936 7884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: Sending DeleteTablet for 3 replicas of tablet b2310d8713a34027b9f18896bcedb2ae
I20260924 05:09:22.999159 8302 tablet_service.cc:1514] Processing DeleteTablet for tablet b2310d8713a34027b9f18896bcedb2ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52576
I20260924 05:09:22.999177 8162 tablet_service.cc:1514] Processing DeleteTablet for tablet b2310d8713a34027b9f18896bcedb2ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58844
I20260924 05:09:22.999248 8420 tablet_replica.cc:330] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: stopping tablet replica
I20260924 05:09:22.999248 8422 tablet_replica.cc:330] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: stopping tablet replica
I20260924 05:09:22.999286 8422 raft_consensus.cc:2229] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:22.999289 8420 raft_consensus.cc:2229] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:22.999258 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet b2310d8713a34027b9f18896bcedb2ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32918
I20260924 05:09:22.999323 8420 raft_consensus.cc:2258] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:22.999347 8421 tablet_replica.cc:330] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: stopping tablet replica
I20260924 05:09:22.999363 8422 raft_consensus.cc:2258] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:22.999392 8421 raft_consensus.cc:2229] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:22.999440 8421 raft_consensus.cc:2258] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:22.999668 8422 ts_tablet_manager.cc:1905] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:22.999718 8421 ts_tablet_manager.cc:1905] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:23.001034 8422 ts_tablet_manager.cc:1918] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:23.001112 8422 log.cc:1196] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/b2310d8713a34027b9f18896bcedb2ae
I20260924 05:09:23.001183 8421 ts_tablet_manager.cc:1918] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:23.001245 8421 log.cc:1196] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/b2310d8713a34027b9f18896bcedb2ae
I20260924 05:09:23.001331 8422 ts_tablet_manager.cc:1939] T b2310d8713a34027b9f18896bcedb2ae P 3a82fdf19149456dbd1da4b702082766: Deleting consensus metadata
I20260924 05:09:23.001477 8421 ts_tablet_manager.cc:1939] T b2310d8713a34027b9f18896bcedb2ae P c87b7f66de404941a0ca95a890e035ad: Deleting consensus metadata
I20260924 05:09:23.001633 7871 catalog_manager.cc:4841] TS 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157): tablet b2310d8713a34027b9f18896bcedb2ae (table test-table [id=ed7481be3e744d238bc1ca2372768bb8]) successfully deleted
I20260924 05:09:23.001838 7872 catalog_manager.cc:4841] TS c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997): tablet b2310d8713a34027b9f18896bcedb2ae (table test-table [id=ed7481be3e744d238bc1ca2372768bb8]) successfully deleted
I20260924 05:09:23.001857 7884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:23.002023 7884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:23.002457 8420 ts_tablet_manager.cc:1905] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:23.003170 8162 tablet_service.cc:1467] Processing CreateTablet for tablet c4e51d2c0a0546e388a14b5351b3e1e1 (DEFAULT_TABLE table=test-table [id=65a4f988d2a74ee49d4661ba47a8a0da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:23.003228 8302 tablet_service.cc:1467] Processing CreateTablet for tablet c4e51d2c0a0546e388a14b5351b3e1e1 (DEFAULT_TABLE table=test-table [id=65a4f988d2a74ee49d4661ba47a8a0da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:23.003327 8302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e51d2c0a0546e388a14b5351b3e1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:23.003372 8162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e51d2c0a0546e388a14b5351b3e1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:23.003599 8022 tablet_service.cc:1467] Processing CreateTablet for tablet c4e51d2c0a0546e388a14b5351b3e1e1 (DEFAULT_TABLE table=test-table [id=65a4f988d2a74ee49d4661ba47a8a0da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:23.003684 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e51d2c0a0546e388a14b5351b3e1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:23.004550 8397 tablet_bootstrap.cc:492] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: Bootstrap starting.
I20260924 05:09:23.004657 8401 tablet_bootstrap.cc:492] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: Bootstrap starting.
I20260924 05:09:23.004940 8397 tablet_bootstrap.cc:654] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:23.004952 8401 tablet_bootstrap.cc:654] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:23.005177 8396 tablet_bootstrap.cc:492] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad: Bootstrap starting.
I20260924 05:09:23.005456 8396 tablet_bootstrap.cc:654] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:23.005686 8420 ts_tablet_manager.cc:1918] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:23.005740 8420 log.cc:1196] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/b2310d8713a34027b9f18896bcedb2ae
I20260924 05:09:23.005915 8401 tablet_bootstrap.cc:492] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: No bootstrap required, opened a new log
I20260924 05:09:23.006000 8401 ts_tablet_manager.cc:1397] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:23.006130 8401 raft_consensus.cc:348] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:23.006234 8401 raft_consensus.cc:374] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:23.006261 8401 raft_consensus.cc:729] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527d8f7059264350a9670377411aeaef, State: Initialized, Role: FOLLOWER
I20260924 05:09:23.006309 8401 consensus_queue.cc:260] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:23.006325 8397 tablet_bootstrap.cc:492] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: No bootstrap required, opened a new log
I20260924 05:09:23.006368 8397 ts_tablet_manager.cc:1397] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:09:23.006398 8401 ts_tablet_manager.cc:1428] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:23.005932 8420 ts_tablet_manager.cc:1939] T b2310d8713a34027b9f18896bcedb2ae P 527d8f7059264350a9670377411aeaef: Deleting consensus metadata
I20260924 05:09:23.006507 8397 raft_consensus.cc:348] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:23.006596 8397 raft_consensus.cc:374] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:23.006624 8397 raft_consensus.cc:729] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a82fdf19149456dbd1da4b702082766, State: Initialized, Role: FOLLOWER
I20260924 05:09:23.006668 8397 consensus_queue.cc:260] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:23.006808 8397 ts_tablet_manager.cc:1428] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:23.007010 7870 catalog_manager.cc:4841] TS 527d8f7059264350a9670377411aeaef (127.0.98.2:35551): tablet b2310d8713a34027b9f18896bcedb2ae (table test-table [id=ed7481be3e744d238bc1ca2372768bb8]) successfully deleted
I20260924 05:09:23.007654 8396 tablet_bootstrap.cc:492] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad: No bootstrap required, opened a new log
I20260924 05:09:23.007705 8396 ts_tablet_manager.cc:1397] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 05:09:23.007809 8396 raft_consensus.cc:348] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:23.007875 8396 raft_consensus.cc:374] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:23.007897 8396 raft_consensus.cc:729] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87b7f66de404941a0ca95a890e035ad, State: Initialized, Role: FOLLOWER
I20260924 05:09:23.007943 8396 consensus_queue.cc:260] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:23.008026 8396 ts_tablet_manager.cc:1428] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:23.076241 8477 raft_consensus.cc:482] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:23.076356 8477 raft_consensus.cc:504] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:23.076604 8477 leader_election.cc:288] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997), 527d8f7059264350a9670377411aeaef (127.0.98.2:35551)
I20260924 05:09:23.076709 8182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e51d2c0a0546e388a14b5351b3e1e1" candidate_uuid: "3a82fdf19149456dbd1da4b702082766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d8f7059264350a9670377411aeaef" is_pre_election: true
I20260924 05:09:23.076716 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e51d2c0a0546e388a14b5351b3e1e1" candidate_uuid: "3a82fdf19149456dbd1da4b702082766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87b7f66de404941a0ca95a890e035ad" is_pre_election: true
I20260924 05:09:23.076807 8182 raft_consensus.cc:2454] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a82fdf19149456dbd1da4b702082766 in term 0.
I20260924 05:09:23.076848 8042 raft_consensus.cc:2454] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a82fdf19149456dbd1da4b702082766 in term 0.
I20260924 05:09:23.076984 8254 leader_election.cc:302] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, 527d8f7059264350a9670377411aeaef; no voters:
I20260924 05:09:23.077080 8403 raft_consensus.cc:2781] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:23.077118 8403 raft_consensus.cc:482] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:23.077148 8403 raft_consensus.cc:3037] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:23.078007 8403 raft_consensus.cc:504] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:23.078181 8403 leader_election.cc:288] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [CANDIDATE]: Term 1 election: Requested vote from peers c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997), 527d8f7059264350a9670377411aeaef (127.0.98.2:35551)
I20260924 05:09:23.078366 8182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e51d2c0a0546e388a14b5351b3e1e1" candidate_uuid: "3a82fdf19149456dbd1da4b702082766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527d8f7059264350a9670377411aeaef"
I20260924 05:09:23.078414 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e51d2c0a0546e388a14b5351b3e1e1" candidate_uuid: "3a82fdf19149456dbd1da4b702082766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87b7f66de404941a0ca95a890e035ad"
I20260924 05:09:23.078446 8182 raft_consensus.cc:3037] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:23.078477 8042 raft_consensus.cc:3037] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:23.079102 8182 raft_consensus.cc:2454] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a82fdf19149456dbd1da4b702082766 in term 1.
I20260924 05:09:23.079252 8042 raft_consensus.cc:2454] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a82fdf19149456dbd1da4b702082766 in term 1.
I20260924 05:09:23.079270 8254 leader_election.cc:302] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a82fdf19149456dbd1da4b702082766, 527d8f7059264350a9670377411aeaef; no voters:
I20260924 05:09:23.079357 8403 raft_consensus.cc:2781] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:23.079399 8403 raft_consensus.cc:686] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 1 LEADER]: Becoming Leader. State: Replica: 3a82fdf19149456dbd1da4b702082766, State: Running, Role: LEADER
I20260924 05:09:23.079457 8403 consensus_queue.cc:237] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } }
I20260924 05:09:23.080034 7885 catalog_manager.cc:5495] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a82fdf19149456dbd1da4b702082766 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3a82fdf19149456dbd1da4b702082766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a82fdf19149456dbd1da4b702082766" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35157 } health_report { overall_health: HEALTHY } } }
I20260924 05:09:23.105125 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:23.102404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35551 (local address 127.0.98.3:45277)
0924 05:09:23.102557 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:23.102592 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:23.102649 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:23.102791 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:23.102791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:23.102793 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:09:23.102864 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:23.102865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:23.103585 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:23.103587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:23.104231 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:23.104233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:23.104641 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:23.104642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:23.104705 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:23.104719 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:09:23.104920 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:09:23.104922 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:09:23.104990 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:09:23.105049 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260924 05:09:23.105115 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:23.102453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45277 (local address 127.0.98.2:35551)
0924 05:09:23.102521 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:23.102524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:23.102637 (+ 113us) server_negotiation.cc:407] Connection header received
0924 05:09:23.102709 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:23.102710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:23.102723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:23.102786 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:23.102941 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:23.103512 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:23.104321 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:23.104574 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:23.104643 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:23.104823 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:23.104855 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:23.104919 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:23.105006 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:09:23.105029 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 05:09:23.107097 8182 raft_consensus.cc:1261] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Refusing update from remote peer 3a82fdf19149456dbd1da4b702082766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:23.107126 8042 raft_consensus.cc:1261] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Refusing update from remote peer 3a82fdf19149456dbd1da4b702082766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:23.107277 8477 consensus_queue.cc:1035] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "527d8f7059264350a9670377411aeaef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:23.107371 8403 consensus_queue.cc:1035] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87b7f66de404941a0ca95a890e035ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:23.114133 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:23.111782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35157 (local address 127.0.98.2:34519)
0924 05:09:23.111818 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:23.111828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:23.111862 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:23.111977 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:23.111977 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:23.111978 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 05:09:23.112014 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:23.112015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:23.112555 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:23.112555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:23.113411 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:23.113414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:23.113764 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:23.113765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:23.113817 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:23.113822 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 05:09:23.113991 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 05:09:23.113992 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 05:09:23.114025 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 05:09:23.114068 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 05:09:23.114320 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:23.111855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34519 (local address 127.0.98.3:35157)
0924 05:09:23.111889 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:23.111891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:23.111898 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:09:23.111915 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:23.111916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:23.111924 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:23.111974 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:23.112059 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:23.112474 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:23.113484 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:23.113689 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:23.113739 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:23.113903 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:23.113934 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:23.113972 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:23.114226 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 05:09:23.114246 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:09:23.120616 7884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:23.120738 7884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:23.121285 7884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09021a56a79342f08e6c68b62fe1fee6: Sending DeleteTablet for 3 replicas of tablet c4e51d2c0a0546e388a14b5351b3e1e1
I20260924 05:09:23.121502 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e51d2c0a0546e388a14b5351b3e1e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32918
I20260924 05:09:23.121523 8302 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e51d2c0a0546e388a14b5351b3e1e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52576
I20260924 05:09:23.121569 8162 tablet_service.cc:1514] Processing DeleteTablet for tablet c4e51d2c0a0546e388a14b5351b3e1e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:23 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58844
I20260924 05:09:23.121665 8420 tablet_replica.cc:330] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: stopping tablet replica
I20260924 05:09:23.121670 8421 tablet_replica.cc:330] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad: stopping tablet replica
I20260924 05:09:23.121722 8420 raft_consensus.cc:2229] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:23.121722 8421 raft_consensus.cc:2229] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:23.121762 8420 raft_consensus.cc:2258] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:23.121762 8421 raft_consensus.cc:2258] T c4e51d2c0a0546e388a14b5351b3e1e1 P c87b7f66de404941a0ca95a890e035ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:23.121851 8422 tablet_replica.cc:330] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: stopping tablet replica
I20260924 05:09:23.121904 8422 raft_consensus.cc:2229] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:23.121976 8422 raft_consensus.cc:2258] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:23.122058 8420 ts_tablet_manager.cc:1905] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:23.122184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 7957
I20260924 05:09:23.122273 8422 ts_tablet_manager.cc:1905] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:23.123618 8420 ts_tablet_manager.cc:1918] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:09:23.123618 8422 ts_tablet_manager.cc:1918] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 05:09:23.123668 8420 log.cc:1196] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/c4e51d2c0a0546e388a14b5351b3e1e1
I20260924 05:09:23.123668 8422 log.cc:1196] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/c4e51d2c0a0546e388a14b5351b3e1e1
I20260924 05:09:23.123878 8420 ts_tablet_manager.cc:1939] T c4e51d2c0a0546e388a14b5351b3e1e1 P 527d8f7059264350a9670377411aeaef: Deleting consensus metadata
I20260924 05:09:23.123878 8422 ts_tablet_manager.cc:1939] T c4e51d2c0a0546e388a14b5351b3e1e1 P 3a82fdf19149456dbd1da4b702082766: Deleting consensus metadata
I20260924 05:09:23.124687 7871 catalog_manager.cc:4841] TS 3a82fdf19149456dbd1da4b702082766 (127.0.98.3:35157): tablet c4e51d2c0a0546e388a14b5351b3e1e1 (table test-table [id=65a4f988d2a74ee49d4661ba47a8a0da]) successfully deleted
I20260924 05:09:23.124866 7870 catalog_manager.cc:4841] TS 527d8f7059264350a9670377411aeaef (127.0.98.2:35551): tablet c4e51d2c0a0546e388a14b5351b3e1e1 (table test-table [id=65a4f988d2a74ee49d4661ba47a8a0da]) successfully deleted
W20260924 05:09:23.128829 7872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37997 (error 108)
I20260924 05:09:23.129074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 8097
I20260924 05:09:23.129742 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:23.129150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37997 (local address 127.0.0.1:32944)
0924 05:09:23.129664 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:23.129703 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":39,"threads_started":1}
W20260924 05:09:23.129920 7872 catalog_manager.cc:4584] TS c87b7f66de404941a0ca95a890e035ad (127.0.98.1:37997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4e51d2c0a0546e388a14b5351b3e1e1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37997: connect: Connection refused (error 111)
I20260924 05:09:23.134680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 8237
I20260924 05:09:23.140089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 7853
2026-09-24T05:09:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2639 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:23 dist-test-slave-mr2f krb5kdc[8492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:23 dist-test-slave-mr2f krb5kdc[8492](info): set up 2 sockets
Sep 24 05:09:23 dist-test-slave-mr2f krb5kdc[8492](info): commencing operation
krb5kdc: starting...
W20260924 05:09:24.175174 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:24 dist-test-slave-mr2f krb5kdc[8492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:09:24.291337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41749
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:24.372891 8505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:24.373077 8505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:24.373098 8505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:24.374187 8505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:24.374224 8505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:24.374238 8505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:24.374251 8505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:24.374300 8505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:24.375820 8505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:24.376046 8505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:24.376273 8505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:24.380759 8511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:24.380837 8513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:24.380797 8510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:24.380952 8505 server_base.cc:1034] running on GCE node
I20260924 05:09:24.381274 8505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:24.381531 8505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:24.382690 8505 hybrid_clock.cc:648] HybridClock initialized: now 1790226564382676 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T05:09:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1248 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:24 dist-test-slave-mr2f krb5kdc[8518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:24 dist-test-slave-mr2f krb5kdc[8518](info): set up 2 sockets
Sep 24 05:09:24 dist-test-slave-mr2f krb5kdc[8518](info): commencing operation
krb5kdc: starting...
W20260924 05:09:25.399462 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:25 dist-test-slave-mr2f krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:09:25.509863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40701
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:25.587261 8531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:25.587495 8531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:25.587530 8531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:25.588671 8531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:25.588737 8531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:25.588761 8531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:25.588788 8531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:25.588819 8531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:25.590286 8531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:25.590546 8531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:25.590775 8531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:25.595381 8539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:25.595393 8537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:25.595393 8536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:25.595628 8531 server_base.cc:1034] running on GCE node
I20260924 05:09:25.595969 8531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:25.596276 8531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:25.597435 8531 hybrid_clock.cc:648] HybridClock initialized: now 1790226565597419 us; error 27 us; skew 500 ppm
Sep 24 05:09:25 dist-test-slave-mr2f krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:25.599908 8531 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:09:25.600339 8531 webserver.cc:460] Webserver started at http://127.0.98.62:37733/ using document root <none> and password file <none>
I20260924 05:09:25.600570 8531 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:25.600637 8531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:25.600788 8531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:25.601809 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "f2b0e989893f426990e2238416a673ac"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "47f9b96a665557128d6ba5e9a075b22f"
server_key_iv: "eecad54139c0d519a4ec946c260a107e"
server_key_version: "encryptionkey@0"
I20260924 05:09:25.602151 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2b0e989893f426990e2238416a673ac"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "47f9b96a665557128d6ba5e9a075b22f"
server_key_iv: "eecad54139c0d519a4ec946c260a107e"
server_key_version: "encryptionkey@0"
I20260924 05:09:25.603549 8531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:25.604307 8546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:25.604514 8531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:25.604606 8531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "f2b0e989893f426990e2238416a673ac"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "47f9b96a665557128d6ba5e9a075b22f"
server_key_iv: "eecad54139c0d519a4ec946c260a107e"
server_key_version: "encryptionkey@0"
I20260924 05:09:25.604687 8531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:25.636919 8531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:25.639338 8531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:25.639506 8531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:25.643249 8531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40701
I20260924 05:09:25.643296 8598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40701 every 8 connection(s)
I20260924 05:09:25.643812 8531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:25.644052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 8531
I20260924 05:09:25.644172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:25.644366 392 external_mini_cluster.cc:1428] Setting key 6dd393404c7f7d38a7418fc38a5f9805
I20260924 05:09:25.644610 8599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:25.647974 8599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: Bootstrap starting.
I20260924 05:09:25.648590 8599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:25.648885 8599 log.cc:824] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:25.649601 8599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: No bootstrap required, opened a new log
Sep 24 05:09:25 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:25.651692 8599 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } }
I20260924 05:09:25.651867 8599 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:25.651906 8599 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b0e989893f426990e2238416a673ac, State: Initialized, Role: FOLLOWER
I20260924 05:09:25.652042 8599 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } }
I20260924 05:09:25.652133 8599 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:25.652180 8599 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:25.652228 8599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:25.652784 8599 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } }
I20260924 05:09:25.652869 8599 leader_election.cc:302] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b0e989893f426990e2238416a673ac; no voters:
I20260924 05:09:25.653057 8599 leader_election.cc:288] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:25.653204 8604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:25.653308 8599 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:25.653399 8604 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 1 LEADER]: Becoming Leader. State: Replica: f2b0e989893f426990e2238416a673ac, State: Running, Role: LEADER
I20260924 05:09:25.653538 8604 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } }
I20260924 05:09:25.653868 8605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2b0e989893f426990e2238416a673ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } }
I20260924 05:09:25.653944 8605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:25.654075 8605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b0e989893f426990e2238416a673ac. Latest consensus state: current_term: 1 leader_uuid: "f2b0e989893f426990e2238416a673ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } }
I20260924 05:09:25.654120 8605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:25.654510 8613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:25.655175 8613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:25.656010 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:25.644895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.0.98.62:40701)
0924 05:09:25.645101 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:25.645110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:25.645141 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:09:25.645819 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:25.645827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:25.645901 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:25.646055 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:25.646610 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.648004 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.648794 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.649362 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.649478 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:25.652098 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:25.652102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:25.652102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:25.652111 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:25.654214 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:25.654695 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:25.654698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:25.654698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:25.654842 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:25.655077 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:25.655078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:25.655078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:25.655372 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:25.655517 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:25.655663 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 05:09:25.655810 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260924 05:09:25.657399 8613 catalog_manager.cc:1347] Generated new cluster ID: 997e761ade6848a1a05c4c705b3f564a
I20260924 05:09:25.657449 8613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:25.667512 8613 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:25.668008 8613 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:25.682228 8613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: Generated new TSK 0
I20260924 05:09:25.682454 8613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:09:25.707847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40701
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:25.786051 8625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:25.786233 8625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:25.786255 8625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:25.787391 8625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:25.787432 8625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:25.787487 8625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:25.788949 8625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:25.789180 8625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:25.789407 8625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:25.794303 8633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:25.794308 8630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:25.794534 8625 server_base.cc:1034] running on GCE node
W20260924 05:09:25.794345 8631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:25.794754 8625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:25.795015 8625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:25.796173 8625 hybrid_clock.cc:648] HybridClock initialized: now 1790226565796159 us; error 28 us; skew 500 ppm
Sep 24 05:09:25 dist-test-slave-mr2f krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:25.798529 8625 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:09:25.799008 8625 webserver.cc:460] Webserver started at http://127.0.98.1:35365/ using document root <none> and password file <none>
I20260924 05:09:25.799239 8625 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:25.799314 8625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:25.799459 8625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:25.800486 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "6eaa18f020034cc7bc24ddbaf3a7aed2"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "f1b273edaf442fcfeb639b1f1e890277"
server_key_iv: "4f50b1f9bb725c1fc4db6f038d151086"
server_key_version: "encryptionkey@0"
I20260924 05:09:25.800828 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6eaa18f020034cc7bc24ddbaf3a7aed2"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "f1b273edaf442fcfeb639b1f1e890277"
server_key_iv: "4f50b1f9bb725c1fc4db6f038d151086"
server_key_version: "encryptionkey@0"
I20260924 05:09:25.802160 8625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:25.802918 8640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:25.803095 8625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:25.803191 8625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "6eaa18f020034cc7bc24ddbaf3a7aed2"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "f1b273edaf442fcfeb639b1f1e890277"
server_key_iv: "4f50b1f9bb725c1fc4db6f038d151086"
server_key_version: "encryptionkey@0"
I20260924 05:09:25.803278 8625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:25.819046 8625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:25.820823 8625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:25.820989 8625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:25.821213 8625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:25.821511 8625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:25.821563 8625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:25.821602 8625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:25.821630 8625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:25.827241 8625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44079
I20260924 05:09:25.827298 8753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44079 every 8 connection(s)
I20260924 05:09:25.827580 8625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:09:25.832294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 8625
I20260924 05:09:25.832409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:25.832665 392 external_mini_cluster.cc:1428] Setting key db9859c7856e05e5c149b13534a3285d
Sep 24 05:09:25 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:25.837160 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:25.828345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40897 (local address 127.0.98.62:40701)
0924 05:09:25.828405 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:25.828408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:25.828620 (+ 212us) server_negotiation.cc:407] Connection header received
0924 05:09:25.829590 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:25.829591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:25.829604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:25.829662 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:25.830248 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.830828 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.831826 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.832049 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.832133 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:25.834329 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:25.834330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:25.834331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:25.834338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:25.835788 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:25.836209 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:25.836210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:25.836211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:25.836279 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:25.836529 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:25.836530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:25.836530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:25.836647 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:25.836722 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:25.836986 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 05:09:25.837070 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:09:25.837365 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:25.828348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:40897)
0924 05:09:25.828504 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:25.828538 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:25.829474 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:25.829732 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:25.829735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:25.829970 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:25.830150 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:25.830154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.830925 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.830926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:25.831720 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:25.831721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.832144 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.832145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:25.832270 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:25.832811 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:25.832819 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:25.834218 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:25.835891 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:25.835897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:25.835899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:25.836116 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:25.836345 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:25.836346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:25.836347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:25.836459 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:25.836714 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:25.836719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:25.836856 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 05:09:25.836967 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 05:09:25.837118 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260924 05:09:25.837878 8754 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40701
I20260924 05:09:25.837955 8754 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:25.838145 8754 heartbeater.cc:511] Master 127.0.98.62:40701 requested a full tablet report, sending...
I20260924 05:09:25.838711 8563 ts_manager.cc:194] Registered new tserver with Master: 6eaa18f020034cc7bc24ddbaf3a7aed2 (127.0.98.1:44079)
I20260924 05:09:25.839665 8563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40897
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:09:25.858289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40701
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:25.937412 8759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:25.937597 8759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:25.937631 8759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:25.938657 8759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:25.938689 8759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:25.938714 8759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:25.940203 8759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:25.940393 8759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:25.940544 8759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:25.945255 8765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:25.945263 8767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:25.945488 8759 server_base.cc:1034] running on GCE node
W20260924 05:09:25.945271 8764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:25.945736 8759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:25.945966 8759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:25.947122 8759 hybrid_clock.cc:648] HybridClock initialized: now 1790226565947111 us; error 28 us; skew 500 ppm
Sep 24 05:09:25 dist-test-slave-mr2f krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:25.949503 8759 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:09:25.949882 8759 webserver.cc:460] Webserver started at http://127.0.98.2:40367/ using document root <none> and password file <none>
I20260924 05:09:25.950129 8759 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:25.950194 8759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:25.950349 8759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:25.951351 8759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "b82ff37624474b57bb22d5dbae7be651"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "673e2311e1ec06e9fabce5480267329d"
server_key_iv: "ecb74f57bb8142f35194086f67b0c88a"
server_key_version: "encryptionkey@0"
I20260924 05:09:25.951687 8759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b82ff37624474b57bb22d5dbae7be651"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "673e2311e1ec06e9fabce5480267329d"
server_key_iv: "ecb74f57bb8142f35194086f67b0c88a"
server_key_version: "encryptionkey@0"
I20260924 05:09:25.952900 8759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:09:25.953678 8774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:25.953860 8759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:25.953951 8759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "b82ff37624474b57bb22d5dbae7be651"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "673e2311e1ec06e9fabce5480267329d"
server_key_iv: "ecb74f57bb8142f35194086f67b0c88a"
server_key_version: "encryptionkey@0"
I20260924 05:09:25.954036 8759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:25.969718 8759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:25.971916 8759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:25.972069 8759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:25.972281 8759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:25.972581 8759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:25.972637 8759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:25.972677 8759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:25.972704 8759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:25.978055 8759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44445
I20260924 05:09:25.978108 8887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44445 every 8 connection(s)
I20260924 05:09:25.978430 8759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:25.982980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 8759
I20260924 05:09:25.983106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:25.983299 392 external_mini_cluster.cc:1428] Setting key 4d14093bcbc62cc3d096cf62284d18b7
Sep 24 05:09:25 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:25.988014 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:25.979164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43991 (local address 127.0.98.62:40701)
0924 05:09:25.979238 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:25.979241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:25.979394 (+ 153us) server_negotiation.cc:407] Connection header received
0924 05:09:25.980245 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:25.980246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:25.980256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:25.980308 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:25.980838 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.981316 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.982227 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.982403 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.982453 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:25.985006 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:25.985007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:25.985007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:25.985013 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:25.986602 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:25.987035 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:25.987037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:25.987037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:25.987099 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:25.987317 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:25.987318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:25.987318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:25.987459 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:25.987532 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:25.987817 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 05:09:25.987908 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 05:09:25.988039 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:25.979116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:43991)
0924 05:09:25.979270 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:25.979303 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:25.980164 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:25.980363 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:25.980366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:25.980599 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:25.980771 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:25.980775 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.981393 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.981393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:25.982158 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:25.982159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:25.982473 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:25.982474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:25.982567 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:25.983104 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:25.983113 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:25.984917 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:25.986680 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:25.986685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:25.986687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:25.986967 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:25.987166 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:25.987167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:25.987167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:25.987257 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:25.987525 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:25.987529 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:25.987654 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 05:09:25.987767 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 05:09:25.987890 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260924 05:09:25.988505 8888 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40701
I20260924 05:09:25.988608 8888 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:25.988770 8888 heartbeater.cc:511] Master 127.0.98.62:40701 requested a full tablet report, sending...
I20260924 05:09:25.989192 8563 ts_manager.cc:194] Registered new tserver with Master: b82ff37624474b57bb22d5dbae7be651 (127.0.98.2:44445)
I20260924 05:09:25.989558 8563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43991
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:09:26.007936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40701
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:26.085974 8893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:26.086197 8893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:26.086234 8893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:26.087390 8893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:26.087460 8893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:26.087505 8893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:26.088955 8893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:26.089226 8893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:26.089453 8893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:26.094501 8898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:26.094538 8893 server_base.cc:1034] running on GCE node
W20260924 05:09:26.094565 8899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:26.094507 8901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:26.094911 8893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:26.095113 8893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:26.096277 8893 hybrid_clock.cc:648] HybridClock initialized: now 1790226566096266 us; error 28 us; skew 500 ppm
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:26.098603 8893 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:09:26.099066 8893 webserver.cc:460] Webserver started at http://127.0.98.3:39543/ using document root <none> and password file <none>
I20260924 05:09:26.099295 8893 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:26.099360 8893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:26.099509 8893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:26.100514 8893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9393a3d05bd46f9b53c74ee878e4a1b"
format_stamp: "Formatted at 2026-09-24 05:09:26 on dist-test-slave-mr2f"
server_key: "1a5df20a918084248645a2d875428409"
server_key_iv: "1878dbd8a8f44bd2a6644e3a4fdf97e7"
server_key_version: "encryptionkey@0"
I20260924 05:09:26.100865 8893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9393a3d05bd46f9b53c74ee878e4a1b"
format_stamp: "Formatted at 2026-09-24 05:09:26 on dist-test-slave-mr2f"
server_key: "1a5df20a918084248645a2d875428409"
server_key_iv: "1878dbd8a8f44bd2a6644e3a4fdf97e7"
server_key_version: "encryptionkey@0"
I20260924 05:09:26.102126 8893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:26.102828 8908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.103003 8893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:26.103096 8893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "f9393a3d05bd46f9b53c74ee878e4a1b"
format_stamp: "Formatted at 2026-09-24 05:09:26 on dist-test-slave-mr2f"
server_key: "1a5df20a918084248645a2d875428409"
server_key_iv: "1878dbd8a8f44bd2a6644e3a4fdf97e7"
server_key_version: "encryptionkey@0"
I20260924 05:09:26.103180 8893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:26.122275 8893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:26.124122 8893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:26.124289 8893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:26.124519 8893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:26.124818 8893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:26.124871 8893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.124922 8893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:26.124953 8893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.131536 8893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33431
I20260924 05:09:26.131736 9021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33431 every 8 connection(s)
I20260924 05:09:26.131949 8893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:26.132205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 8893
I20260924 05:09:26.132319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:26.132490 392 external_mini_cluster.cc:1428] Setting key 3077d820bbaaae0eac6f88f25f68ae23
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:26.140622 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.132661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50599 (local address 127.0.98.62:40701)
0924 05:09:26.132711 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.132714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.132895 (+ 181us) server_negotiation.cc:407] Connection header received
0924 05:09:26.133999 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.133999 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.134010 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.134060 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.134628 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.135205 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.136151 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.136334 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.136384 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.138289 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:26.138290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:26.138291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:26.138297 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:26.139561 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:26.139895 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:26.139896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:26.139896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:26.139944 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:26.140152 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:26.140153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:26.140153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:26.140243 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:26.140289 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:26.140505 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 05:09:26.140561 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:09:26.140789 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.132587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:50599)
0924 05:09:26.132784 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.132824 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.133899 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.134112 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.134115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.134358 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.134564 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.134567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.135287 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.135288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.136083 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.136085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.136410 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.136410 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.136504 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.136909 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.136917 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.138211 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:26.139629 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:26.139634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:26.139635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:26.139834 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:26.139999 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:26.140000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:26.140000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:26.140090 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:26.140293 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:26.140296 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:26.140404 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 05:09:26.140479 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:09:26.140584 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260924 05:09:26.141206 9022 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40701
I20260924 05:09:26.141304 9022 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:26.141472 9022 heartbeater.cc:511] Master 127.0.98.62:40701 requested a full tablet report, sending...
I20260924 05:09:26.141896 8563 ts_manager.cc:194] Registered new tserver with Master: f9393a3d05bd46f9b53c74ee878e4a1b (127.0.98.3:33431)
I20260924 05:09:26.142308 8563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50599
I20260924 05:09:26.144110 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 05:09:26.212687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 8625
I20260924 05:09:26.217446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 8759
I20260924 05:09:26.221766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 8893
I20260924 05:09:26.225636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 8531
I20260924 05:09:26.230440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40701
--webserver_interface=127.0.98.62
--webserver_port=37733
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:26.312366 9031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:26.312564 9031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:26.312589 9031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:26.313719 9031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:26.313758 9031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:26.313772 9031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:26.313786 9031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:26.313829 9031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:26.315346 9031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:26.315586 9031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:26.315826 9031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:26.320546 9037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:26.320559 9039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:26.320559 9036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:26.320684 9031 server_base.cc:1034] running on GCE node
I20260924 05:09:26.320986 9031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:26.321208 9031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:26.322397 9031 hybrid_clock.cc:648] HybridClock initialized: now 1790226566322381 us; error 32 us; skew 500 ppm
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:26.324910 9031 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:09:26.325321 9031 webserver.cc:460] Webserver started at http://127.0.98.62:37733/ using document root <none> and password file <none>
I20260924 05:09:26.325554 9031 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:26.325609 9031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:26.327040 9031 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:26.327697 9046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.327881 9031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.327992 9031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "f2b0e989893f426990e2238416a673ac"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "47f9b96a665557128d6ba5e9a075b22f"
server_key_iv: "eecad54139c0d519a4ec946c260a107e"
server_key_version: "encryptionkey@0"
I20260924 05:09:26.328315 9031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:26.345799 9031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:26.347788 9031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:26.347934 9031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:26.351611 9031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40701
I20260924 05:09:26.351656 9098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40701 every 8 connection(s)
I20260924 05:09:26.352187 9031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:26.353385 9099 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:09:26.353997 9099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: Bootstrap starting.
I20260924 05:09:26.354959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 9031
I20260924 05:09:26.355252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44079
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40701
--webserver_port=35365
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:09:26.362001 9099 log.cc:824] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:26.363713 9099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:09:26.363991 9099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: Bootstrap complete.
I20260924 05:09:26.366117 9099 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } }
I20260924 05:09:26.366356 9099 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b0e989893f426990e2238416a673ac, State: Initialized, Role: FOLLOWER
I20260924 05:09:26.366573 9099 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } }
I20260924 05:09:26.366670 9099 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:26.366722 9099 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:26.366789 9099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:09:26.367759 9099 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } }
I20260924 05:09:26.367882 9099 leader_election.cc:302] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b0e989893f426990e2238416a673ac; no voters:
I20260924 05:09:26.368077 9099 leader_election.cc:288] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:09:26.368109 9103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:09:26.368213 9103 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [term 2 LEADER]: Becoming Leader. State: Replica: f2b0e989893f426990e2238416a673ac, State: Running, Role: LEADER
I20260924 05:09:26.368357 9099 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:26.368361 9103 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } }
I20260924 05:09:26.368686 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f2b0e989893f426990e2238416a673ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } }
I20260924 05:09:26.368770 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:26.369006 9105 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b0e989893f426990e2238416a673ac. Latest consensus state: current_term: 2 leader_uuid: "f2b0e989893f426990e2238416a673ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0e989893f426990e2238416a673ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } }
I20260924 05:09:26.369103 9105 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:26.369341 9109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:26.369910 9109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:26.370034 9109 catalog_manager.cc:1259] Loaded cluster ID: 997e761ade6848a1a05c4c705b3f564a
I20260924 05:09:26.370066 9109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:26.371454 9109 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:26.371748 9109 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f2b0e989893f426990e2238416a673ac: Loaded TSK: 0
I20260924 05:09:26.371984 9109 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:09:26.440963 9101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:26.441157 9101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:26.441180 9101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:26.442337 9101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:26.442377 9101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:26.442402 9101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:26.443868 9101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35365
--tserver_master_addrs=127.0.98.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:26.444108 9101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:26.444350 9101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:26.450093 9125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:26.450121 9126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:26.450133 9128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:26.450350 9101 server_base.cc:1034] running on GCE node
I20260924 05:09:26.450528 9101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:26.450853 9101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:26.452004 9101 hybrid_clock.cc:648] HybridClock initialized: now 1790226566451994 us; error 25 us; skew 500 ppm
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:26.454635 9101 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 05:09:26.455165 9101 webserver.cc:460] Webserver started at http://127.0.98.1:35365/ using document root <none> and password file <none>
I20260924 05:09:26.455436 9101 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:26.455518 9101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:26.457223 9101 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:09:26.457870 9135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.458084 9101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:26.458175 9101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "6eaa18f020034cc7bc24ddbaf3a7aed2"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "f1b273edaf442fcfeb639b1f1e890277"
server_key_iv: "4f50b1f9bb725c1fc4db6f038d151086"
server_key_version: "encryptionkey@0"
I20260924 05:09:26.458606 9101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:26.470961 9101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:26.473196 9101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:26.473438 9101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:26.473670 9101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:26.473966 9101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:26.474031 9101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.474072 9101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:26.474100 9101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.479673 9101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44079
I20260924 05:09:26.479719 9248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44079 every 8 connection(s)
I20260924 05:09:26.480075 9101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:26.489457 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.480788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:43569)
0924 05:09:26.480992 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.481036 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.481660 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.482079 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.482084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.482845 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.483120 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.483126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.484243 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.484244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.485508 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.485511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.486075 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.486076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.486214 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.486740 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.486749 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.489248 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":116,"threads_started":1}
W20260924 05:09:26.489687 9249 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40701 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40701: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 05:09:26.489924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 9101
W20260924 05:09:26.490026 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:26.480947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43569 (local address 127.0.98.62:40701)
0924 05:09:26.481156 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.481164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.481202 (+ 38us) server_negotiation.cc:407] Connection header received
0924 05:09:26.481834 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.481840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.481877 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.481985 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.483236 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.484136 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.485596 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.485981 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.486057 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.489373 (+ 3316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43569: BlockingRecv error: recv got EOF from 127.0.98.1:43569 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":130,"threads_started":1}
I20260924 05:09:26.490257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44445
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40701
--webserver_port=40367
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:26.495406 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.489926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:43019)
0924 05:09:26.490143 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.490155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.490196 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.490439 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.490439 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.490618 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.490673 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.490675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.491619 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.491620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.492411 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.492414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.492912 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.492913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.492975 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.493444 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.493449 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.495340 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199}
W20260924 05:09:26.495702 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:26.489993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43019 (local address 127.0.98.62:40701)
0924 05:09:26.490305 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.490307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.490314 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:09:26.490336 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.490337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.490349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.490398 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.490786 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.491514 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.492511 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.492807 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.492863 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.495646 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43019: BlockingRecv error: recv got EOF from 127.0.98.1:43019 (error 108)
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":91,"threads_started":1}
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:26.501442 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.496185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:57979)
0924 05:09:26.496235 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.496245 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.496283 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.496436 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.496437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.496593 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.496630 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.496632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.497506 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.497507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.498411 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.498414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.498885 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.498886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.498968 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.499443 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.499448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.501374 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 05:09:26.501514 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:26.496241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57979 (local address 127.0.98.62:40701)
0924 05:09:26.496277 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.496279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.496288 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:26.496345 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.496345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.496354 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.496412 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.496710 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.497417 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.498503 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.498813 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.498880 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.501455 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57979: BlockingRecv error: recv got EOF from 127.0.98.1:57979 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 05:09:26.501771 9249 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:09:26.570664 9254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:26.570899 9254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:26.570940 9254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:26.572101 9254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:26.572165 9254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:26.572211 9254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:26.573662 9254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40367
--tserver_master_addrs=127.0.98.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:26.573917 9254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:26.574141 9254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:26.579043 9262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:26.579071 9254 server_base.cc:1034] running on GCE node
W20260924 05:09:26.579097 9259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:26.579269 9260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:26.579470 9254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:26.579712 9254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:26.580852 9254 hybrid_clock.cc:648] HybridClock initialized: now 1790226566580838 us; error 29 us; skew 500 ppm
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:26.583247 9254 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 05:09:26.583711 9254 webserver.cc:460] Webserver started at http://127.0.98.2:40367/ using document root <none> and password file <none>
I20260924 05:09:26.583930 9254 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:26.584015 9254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:26.585287 9254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:26.585858 9269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.586026 9254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.586122 9254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "b82ff37624474b57bb22d5dbae7be651"
format_stamp: "Formatted at 2026-09-24 05:09:25 on dist-test-slave-mr2f"
server_key: "673e2311e1ec06e9fabce5480267329d"
server_key_iv: "ecb74f57bb8142f35194086f67b0c88a"
server_key_version: "encryptionkey@0"
I20260924 05:09:26.586444 9254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:26.593343 9254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:26.595273 9254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:26.595427 9254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:26.595649 9254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:26.595939 9254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:26.596000 9254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.596040 9254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:26.596068 9254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.602010 9254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44445
I20260924 05:09:26.602048 9382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44445 every 8 connection(s)
I20260924 05:09:26.602352 9254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:26.604825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 9254
I20260924 05:09:26.605142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33431
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40701
--webserver_port=39543
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 05:09:26.610147 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:26.603012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41271 (local address 127.0.98.62:40701)
0924 05:09:26.603066 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.603067 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.603251 (+ 184us) server_negotiation.cc:407] Connection header received
0924 05:09:26.603668 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.603669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.603681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.603731 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.604533 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.605052 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.606540 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.606903 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.606975 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.610102 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41271: BlockingRecv error: recv got EOF from 127.0.98.2:41271 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:09:26.610229 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.603000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:41271)
0924 05:09:26.603134 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.603164 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.603593 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.603777 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.603780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.604285 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.604465 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.604470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.605143 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.605144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.606420 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.606423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.607009 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.607010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.607114 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.607609 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.607616 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.610003 (+ 2387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
W20260924 05:09:26.610489 9383 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40701 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40701: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:26.615414 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.610695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:33519)
0924 05:09:26.610748 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.610758 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.610831 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.611027 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.611028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.611159 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.611209 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.611210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.612180 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.612181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.612668 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.612669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.613164 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.613165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.613211 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.613666 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.613670 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.615373 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 05:09:26.615586 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:26.610752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33519 (local address 127.0.98.62:40701)
0924 05:09:26.610816 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.610818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.610827 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:26.610898 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.610899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.610908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.610983 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.611297 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.612093 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.612752 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.613086 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.613165 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.615493 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33519: BlockingRecv error: recv got EOF from 127.0.98.2:33519 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:26.620236 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.615657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:52635)
0924 05:09:26.615804 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.615816 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.615870 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.615992 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.615993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.616186 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.616251 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.616253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.616825 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.616826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.617711 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.617713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.618053 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.618054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.618112 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.618493 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.618497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.620182 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
W20260924 05:09:26.620318 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:26.615860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52635 (local address 127.0.98.62:40701)
0924 05:09:26.615895 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.615897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.615913 (+ 16us) server_negotiation.cc:407] Connection header received
0924 05:09:26.615931 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.615931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.615938 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.615983 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.616328 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.616749 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.617794 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.617964 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.618019 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.620267 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52635: BlockingRecv error: recv got EOF from 127.0.98.2:52635 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 05:09:26.620357 9383 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:09:26.688122 9386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:26.688355 9386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:26.688390 9386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:26.689519 9386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:26.689587 9386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:26.689632 9386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:26.691071 9386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39543
--tserver_master_addrs=127.0.98.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:26.691351 9386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:26.691572 9386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:26.696369 9393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:26.696375 9395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:26.696445 9386 server_base.cc:1034] running on GCE node
W20260924 05:09:26.696379 9392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:26.696751 9386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:26.697036 9386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:26.698201 9386 hybrid_clock.cc:648] HybridClock initialized: now 1790226566698190 us; error 26 us; skew 500 ppm
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:26.700502 9386 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 05:09:26.700946 9386 webserver.cc:460] Webserver started at http://127.0.98.3:39543/ using document root <none> and password file <none>
I20260924 05:09:26.701189 9386 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:26.701269 9386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:26.702544 9386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:26.703114 9402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.703284 9386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.703377 9386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "f9393a3d05bd46f9b53c74ee878e4a1b"
format_stamp: "Formatted at 2026-09-24 05:09:26 on dist-test-slave-mr2f"
server_key: "1a5df20a918084248645a2d875428409"
server_key_iv: "1878dbd8a8f44bd2a6644e3a4fdf97e7"
server_key_version: "encryptionkey@0"
I20260924 05:09:26.703692 9386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:26.719996 9386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:26.721766 9386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:26.721928 9386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:26.722159 9386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:26.722466 9386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:26.722520 9386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.722561 9386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:26.722589 9386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:26.728384 9386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33431
I20260924 05:09:26.728427 9515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33431 every 8 connection(s)
I20260924 05:09:26.728747 9386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:26.730106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 9386
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 05:09:26.737056 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:26.729430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43545 (local address 127.0.98.62:40701)
0924 05:09:26.729516 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.729519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.729698 (+ 179us) server_negotiation.cc:407] Connection header received
0924 05:09:26.730362 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.730367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.730378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.730413 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.731791 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.732238 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.733364 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.733642 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.733696 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.736998 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43545: BlockingRecv error: recv got EOF from 127.0.98.3:43545 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 05:09:26.737123 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.729395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:43545)
0924 05:09:26.729579 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.729614 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.730242 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.730830 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.730835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.731461 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.731704 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.731709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.732354 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.732355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.733270 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.733272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.733723 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.733724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.733917 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.734379 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.734386 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.736935 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
W20260924 05:09:26.737308 9516 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40701 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40701: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 05:09:26.738041 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.730651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.0.98.62:40701)
0924 05:09:26.730705 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.730707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.730821 (+ 114us) server_negotiation.cc:407] Connection header received
0924 05:09:26.730893 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.730894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.730909 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.730984 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.731620 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.732217 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.732995 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.733281 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.733350 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.734575 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:26.734580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:26.734580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:26.734590 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:26.736439 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:26.736840 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:26.736843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:26.736843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:26.736930 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:26.737182 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:26.737183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:26.737183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:26.737469 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:26.737594 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:26.737844 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 05:09:26.737951 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:26.741556 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.737528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:33745)
0924 05:09:26.737559 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.737568 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.737628 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.737760 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.737760 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.737891 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.737930 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.737931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.738813 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.738814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.739352 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.739354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.739649 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.739650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.739693 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.740030 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.740034 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.741517 (+ 1483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 05:09:26.741664 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:26.737580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33745 (local address 127.0.98.62:40701)
0924 05:09:26.737638 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.737640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.737650 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:26.737681 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.737682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.737690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.737757 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.738087 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.738697 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.739417 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.739587 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.739636 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.741594 (+ 1958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33745: BlockingRecv error: recv got EOF from 127.0.98.3:33745 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:26 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:26.745503 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:26.741870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:41665)
0924 05:09:26.741909 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:26.741920 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:26.741956 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:26.742079 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:26.742080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:26.742195 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:26.742217 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.742218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.742862 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.742863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.743331 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:26.743332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.743690 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.743691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:26.743742 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.744054 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:26.744057 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:26.745463 (+ 1406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:09:26.745654 9516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:09:26.745676 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:26.741905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41665 (local address 127.0.98.62:40701)
0924 05:09:26.741958 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:26.741960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:26.741969 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:26.742008 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:26.742009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:26.742018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:26.742070 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:26.742313 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.742744 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.743427 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:26.743608 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:26.743689 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:26.745612 (+ 1923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41665: BlockingRecv error: recv got EOF from 127.0.98.3:41665 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 05:09:27 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:27 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:27.515133 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:27.502276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:59425)
0924 05:09:27.502427 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:27.502438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:27.502490 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:27.502622 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:27.502623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:27.502836 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:27.502881 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:27.502883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.503498 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.503498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:27.504022 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:27.504024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.504401 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.504402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:27.504470 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:27.513285 (+ 8815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:27.513291 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:27.515066 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
W20260924 05:09:27.515275 9521 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:27.502369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59425 (local address 127.0.98.62:40701)
0924 05:09:27.502471 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:27.502473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:27.502481 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:27.502547 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:27.502547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:27.502556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:27.502603 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:27.502948 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.503436 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.504088 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.504330 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.504400 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:27.515171 (+ 10771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59425: BlockingRecv error: recv got EOF from 127.0.98.1:59425 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
Sep 24 05:09:27 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:27 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:27.625316 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:27.620719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:50359)
0924 05:09:27.620864 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:27.620874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:27.620936 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:27.621071 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:27.621072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:27.621219 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:27.621264 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:27.621266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.621880 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.621881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:27.622395 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:27.622396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.622814 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.622816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:27.622909 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:27.623588 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:27.623592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:27.625223 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
W20260924 05:09:27.625351 9521 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:27.620818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50359 (local address 127.0.98.62:40701)
0924 05:09:27.620887 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:27.620890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:27.620935 (+ 45us) server_negotiation.cc:407] Connection header received
0924 05:09:27.620978 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:27.620978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:27.620996 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:27.621043 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:27.621321 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.621811 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.622460 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.622664 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.622816 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:27.625295 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50359: BlockingRecv error: recv got EOF from 127.0.98.2:50359 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 05:09:27 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:27 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:27.750685 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:27.746113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:54515)
0924 05:09:27.746294 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:27.746305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:27.746373 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:27.746536 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:27.746537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:27.746711 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:27.746753 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:27.746779 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.747463 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.747464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:27.748011 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:27.748012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.748383 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.748384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:27.748444 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:27.748948 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:27.748952 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:27.750585 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
W20260924 05:09:27.750723 9521 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:27.746208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54515 (local address 127.0.98.62:40701)
0924 05:09:27.746275 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:27.746278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:27.746355 (+ 77us) server_negotiation.cc:407] Connection header received
0924 05:09:27.746427 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:27.746428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:27.746461 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:27.746527 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:27.746893 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.747380 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.748099 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:27.748289 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:27.748380 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:27.750659 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54515: BlockingRecv error: recv got EOF from 127.0.98.3:54515 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 05:09:28 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:28 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:28.520351 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:28.515623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:49001)
0924 05:09:28.515802 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:28.515813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:28.515913 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:28.516060 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:28.516061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:28.516244 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:28.516324 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:28.516327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.517039 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.517040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:28.517574 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:28.517576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.517950 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.517951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:28.518018 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:28.518485 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:28.518491 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:28.520255 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
W20260924 05:09:28.520402 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:28.515719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49001 (local address 127.0.98.62:40701)
0924 05:09:28.515839 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:28.515841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:28.515936 (+ 95us) server_negotiation.cc:407] Connection header received
0924 05:09:28.515964 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:28.515965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:28.515977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:28.516054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:28.516395 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.516949 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.517662 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.517863 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.517950 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:28.520331 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49001: BlockingRecv error: recv got EOF from 127.0.98.1:49001 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
Sep 24 05:09:28 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:28 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:28.630334 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:28.625921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:44557)
0924 05:09:28.626082 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:28.626093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:28.626158 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:28.626284 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:28.626285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:28.626445 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:28.626492 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:28.626494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.627173 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.627174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:28.627718 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:28.627721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.628025 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.628026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:28.628065 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:28.628578 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:28.628583 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:28.630228 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
W20260924 05:09:28.630388 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:28.626010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44557 (local address 127.0.98.62:40701)
0924 05:09:28.626062 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:28.626064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:28.626163 (+ 99us) server_negotiation.cc:407] Connection header received
0924 05:09:28.626203 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:28.626203 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:28.626212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:28.626265 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:28.626554 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.627091 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.627790 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.627964 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.628018 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:28.630312 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44557: BlockingRecv error: recv got EOF from 127.0.98.2:44557 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 05:09:28 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:28 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:28.755959 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:28.751224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:57757)
0924 05:09:28.751360 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:28.751377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:28.751451 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:28.751598 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:28.751599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:28.751769 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:28.751822 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:28.751824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.752637 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.752639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:28.753259 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:28.753262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.753639 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.753640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:28.753678 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:28.754094 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:28.754098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:28.755901 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
W20260924 05:09:28.756054 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:28.751396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57757 (local address 127.0.98.62:40701)
0924 05:09:28.751459 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:28.751462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:28.751472 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:28.751503 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:28.751504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:28.751519 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:28.751592 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:28.751903 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.752548 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.753378 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:28.753564 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:28.753628 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:28.755987 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57757: BlockingRecv error: recv got EOF from 127.0.98.3:57757 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 05:09:29 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:29 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:29.525254 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:29.520867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:53341)
0924 05:09:29.521002 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:29.521015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:29.521077 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:29.521212 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:29.521212 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:29.521369 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:29.521416 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:29.521417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.522028 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.522029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:29.522551 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:29.522552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.522879 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.522880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:29.522918 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:29.523303 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:29.523307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:29.525153 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
W20260924 05:09:29.525297 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:29.520912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53341 (local address 127.0.98.62:40701)
0924 05:09:29.521060 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:29.521062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:29.521070 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:29.521132 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:29.521132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:29.521142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:29.521191 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:29.521476 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.521966 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.522615 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.522811 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.522876 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:29.525229 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53341: BlockingRecv error: recv got EOF from 127.0.98.1:53341 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
Sep 24 05:09:29 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:29 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:29.635143 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:29.630870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:51247)
0924 05:09:29.631103 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:29.631115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:29.631176 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:29.631286 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:29.631286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:29.631437 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:29.631483 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:29.631485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.632136 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.632137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:29.632632 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:29.632633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.632938 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.632939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:29.632974 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:29.633410 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:29.633414 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:29.635064 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":178,"threads_started":1}
W20260924 05:09:29.635200 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:29.630894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51247 (local address 127.0.98.62:40701)
0924 05:09:29.631061 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:29.631064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:29.631177 (+ 113us) server_negotiation.cc:407] Connection header received
0924 05:09:29.631222 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:29.631222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:29.631232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:29.631275 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:29.631551 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.632072 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.632701 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.632877 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.632926 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:29.635158 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51247: BlockingRecv error: recv got EOF from 127.0.98.2:51247 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Sep 24 05:09:29 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:29 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:29.761059 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:29.756441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:52875)
0924 05:09:29.756615 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:29.756632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:29.756707 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:29.756864 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:29.756865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:29.757043 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:29.757089 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:29.757091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.757743 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.757744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:29.758271 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:29.758272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.758653 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.758654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:29.758720 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:29.759211 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:29.759216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:29.760957 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
W20260924 05:09:29.761106 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:29.756503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52875 (local address 127.0.98.62:40701)
0924 05:09:29.756573 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:29.756575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:29.756687 (+ 112us) server_negotiation.cc:407] Connection header received
0924 05:09:29.756769 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:29.756770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:29.756783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:29.756858 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:29.757169 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.757661 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.758371 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:29.758566 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:29.758651 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:29.761040 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52875: BlockingRecv error: recv got EOF from 127.0.98.3:52875 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 05:09:30 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:30 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:30.530062 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:30.525699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:58373)
0924 05:09:30.525817 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:30.525828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:30.525892 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:30.526047 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:30.526048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:30.526197 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:30.526246 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:30.526247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.526887 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.526887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:30.527397 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:30.527398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.527708 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.527709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:30.527747 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:30.528280 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:30.528290 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:30.530007 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
W20260924 05:09:30.530224 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:30.525781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58373 (local address 127.0.98.62:40701)
0924 05:09:30.525896 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:30.525898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:30.525909 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:09:30.525940 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:30.525940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:30.525949 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:30.526021 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:30.526309 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.526816 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.527464 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.527647 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.527696 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:30.530086 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58373: BlockingRecv error: recv got EOF from 127.0.98.1:58373 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
Sep 24 05:09:30 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:30 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:30.640203 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:30.635595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:58139)
0924 05:09:30.635765 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:30.635785 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:30.635889 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:30.636021 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:30.636022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:30.636201 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:30.636244 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:30.636246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.636915 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.636916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:30.637442 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:30.637444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.637822 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.637823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:30.637880 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:30.638375 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:30.638379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:30.640146 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
W20260924 05:09:30.640357 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:30.635680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58139 (local address 127.0.98.62:40701)
0924 05:09:30.635753 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:30.635755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:30.635875 (+ 120us) server_negotiation.cc:407] Connection header received
0924 05:09:30.635947 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:30.635948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:30.635961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:30.636013 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:30.636327 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.636828 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.637540 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.637736 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.637820 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:30.640273 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58139: BlockingRecv error: recv got EOF from 127.0.98.2:58139 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 05:09:30 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:30 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:30.766227 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:30.761655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:43163)
0924 05:09:30.761825 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:30.761836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:30.761892 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:30.762109 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:30.762110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:30.762259 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:30.762307 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:30.762308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.762970 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.762971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:30.763557 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:30.763559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.763904 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.763904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:30.763942 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:30.764479 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:30.764483 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:30.766169 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
W20260924 05:09:30.766402 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:30.761731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43163 (local address 127.0.98.62:40701)
0924 05:09:30.761849 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:30.761851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:30.761878 (+ 27us) server_negotiation.cc:407] Connection header received
0924 05:09:30.762007 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:30.762008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:30.762024 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:30.762094 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:30.762372 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.762906 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.763634 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:30.763840 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:30.763895 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:30.766266 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43163: BlockingRecv error: recv got EOF from 127.0.98.3:43163 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 05:09:31 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:31 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:31.535251 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:31.530546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:37455)
0924 05:09:31.530690 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:31.530703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:31.530808 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:31.530998 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:31.530999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:31.531171 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:31.531216 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:31.531218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.531983 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.531984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:31.532505 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:31.532507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.532908 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.532908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:31.532971 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:31.533441 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:31.533453 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:31.535159 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
W20260924 05:09:31.535300 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:31.530720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37455 (local address 127.0.98.62:40701)
0924 05:09:31.530867 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:31.530869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:31.530883 (+ 14us) server_negotiation.cc:407] Connection header received
0924 05:09:31.530905 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:31.530906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:31.530915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:31.530990 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:31.531297 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.531902 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.532599 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.532825 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.532905 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:31.535232 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37455: BlockingRecv error: recv got EOF from 127.0.98.1:37455 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
Sep 24 05:09:31 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:31 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:31.645004 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:31.640713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:51441)
0924 05:09:31.640865 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:31.640897 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:31.640949 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:31.641081 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:31.641082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:31.641238 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:31.641285 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:31.641287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.641897 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.641898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:31.642410 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:31.642412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.642715 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.642716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:31.642752 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:31.643191 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:31.643195 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:31.644901 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
W20260924 05:09:31.645025 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:31.640845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51441 (local address 127.0.98.62:40701)
0924 05:09:31.640906 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:31.640908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:31.640936 (+ 28us) server_negotiation.cc:407] Connection header received
0924 05:09:31.641007 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:31.641008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:31.641018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:31.641059 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:31.641346 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.641830 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.642479 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.642653 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.642702 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:31.644966 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51441: BlockingRecv error: recv got EOF from 127.0.98.2:51441 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 05:09:31 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:31 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:31.771183 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:31.766678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:60689)
0924 05:09:31.766822 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:31.766833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:31.766901 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:31.767026 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:31.767026 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:31.767180 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:31.767226 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:31.767227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.767834 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.767835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:31.768355 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:31.768356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.768741 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.768742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:31.768780 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:31.769277 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:31.769282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:31.771089 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
W20260924 05:09:31.771222 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:31.766747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60689 (local address 127.0.98.62:40701)
0924 05:09:31.766834 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:31.766835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:31.766906 (+ 71us) server_negotiation.cc:407] Connection header received
0924 05:09:31.766943 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:31.766943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:31.766954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:31.767005 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:31.767285 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.767768 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.768437 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:31.768671 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:31.768740 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:31.771160 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60689: BlockingRecv error: recv got EOF from 127.0.98.3:60689 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 05:09:32 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:32 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:32.548983 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:32.535686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:43995)
0924 05:09:32.535836 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:32.535846 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:32.535906 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:32.536025 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:32.536026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:32.536177 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:32.536223 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:32.536225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.536866 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.536867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:32.537413 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:32.537416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.537741 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.537742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:32.537779 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:32.547210 (+ 9431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:32.547215 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:32.548915 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
W20260924 05:09:32.549088 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:32.535766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43995 (local address 127.0.98.62:40701)
0924 05:09:32.535877 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:32.535879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:32.535897 (+ 18us) server_negotiation.cc:407] Connection header received
0924 05:09:32.535962 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:32.535963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:32.535973 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:32.536020 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:32.536282 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.536775 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.537488 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.537677 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.537725 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:32.549012 (+ 11287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43995: BlockingRecv error: recv got EOF from 127.0.98.1:43995 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
Sep 24 05:09:32 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:32 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:32.649964 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:32.645482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:45945)
0924 05:09:32.645625 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:32.645634 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:32.645683 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:32.645829 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:32.645829 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:32.645987 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:32.646034 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:32.646035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.646645 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.646646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:32.647243 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:32.647247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.647560 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.647561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:32.647596 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:32.648023 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:32.648028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:32.649845 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
W20260924 05:09:32.649992 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:32.645617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45945 (local address 127.0.98.62:40701)
0924 05:09:32.645701 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:32.645703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:32.645714 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:09:32.645746 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:32.645747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:32.645756 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:32.645814 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:32.646098 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.646569 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.647316 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.647491 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.647549 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:32.649932 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45945: BlockingRecv error: recv got EOF from 127.0.98.2:45945 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 05:09:32 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:32 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:32.776469 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:32.771643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:46235)
0924 05:09:32.771917 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:32.771928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:32.772022 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:32.772195 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:32.772197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:32.772384 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:32.772428 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:32.772429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.773115 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.773116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:32.773628 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:32.773630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.773989 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.773990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:32.774054 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:32.774625 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:32.774629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:32.776353 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":205,"threads_started":1}
W20260924 05:09:32.776497 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:32.771825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46235 (local address 127.0.98.62:40701)
0924 05:09:32.771912 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:32.771915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:32.772007 (+ 92us) server_negotiation.cc:407] Connection header received
0924 05:09:32.772108 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:32.772109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:32.772124 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:32.772190 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:32.772511 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.773030 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.773713 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:32.773905 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:32.773988 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:32.776433 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46235: BlockingRecv error: recv got EOF from 127.0.98.3:46235 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 05:09:33 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:33 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:33.553996 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:33.549558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:47965)
0924 05:09:33.549721 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:33.549731 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:33.549788 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:33.549932 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:33.549932 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:33.550088 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:33.550137 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:33.550139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.550817 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.550820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:33.551351 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:33.551353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.551709 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.551710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:33.551750 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:33.552295 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:33.552300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:33.553939 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
W20260924 05:09:33.554198 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:33.549623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47965 (local address 127.0.98.62:40701)
0924 05:09:33.549745 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:33.549747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:33.549781 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:09:33.549846 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:33.549847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:33.549858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:33.549910 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:33.550200 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.550703 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.551424 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.551626 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.551696 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:33.554103 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47965: BlockingRecv error: recv got EOF from 127.0.98.1:47965 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
Sep 24 05:09:33 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:33 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:33.655206 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:33.650518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:39965)
0924 05:09:33.650658 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:33.650673 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:33.650732 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:33.650940 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:33.650941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:33.651106 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:33.651156 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:33.651158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.651799 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.651800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:33.652346 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:33.652348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.652777 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.652778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:33.652820 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:33.653291 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:33.653295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:33.655147 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
W20260924 05:09:33.655318 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:33.650590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39965 (local address 127.0.98.62:40701)
0924 05:09:33.650676 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:33.650678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:33.650721 (+ 43us) server_negotiation.cc:407] Connection header received
0924 05:09:33.650850 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:33.650851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:33.650862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:33.650920 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:33.651222 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.651704 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.652436 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.652708 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.652763 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:33.655241 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39965: BlockingRecv error: recv got EOF from 127.0.98.2:39965 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 05:09:33 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:33 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:33.781524 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:33.777087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:49287)
0924 05:09:33.777224 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:33.777234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:33.777305 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:33.777432 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:33.777432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:33.777589 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:33.777638 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:33.777639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.778248 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.778249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:33.778833 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:33.778836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.779155 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.779156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:33.779195 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:33.779726 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:33.779731 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:33.781466 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
W20260924 05:09:33.781629 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:33.777153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49287 (local address 127.0.98.62:40701)
0924 05:09:33.777201 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:33.777203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:33.777296 (+ 93us) server_negotiation.cc:407] Connection header received
0924 05:09:33.777357 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:33.777358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:33.777369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:33.777409 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:33.777700 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.778182 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.778906 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:33.779092 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:33.779144 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:33.781559 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49287: BlockingRecv error: recv got EOF from 127.0.98.3:49287 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 05:09:34 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:34 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:34.558825 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:34.554456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:39225)
0924 05:09:34.554604 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:34.554616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:34.554683 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:34.554898 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:34.554899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:34.555060 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:34.555110 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:34.555112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.555747 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.555748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:34.556280 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:34.556282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.556610 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.556611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:34.556651 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:34.557092 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:34.557097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:34.558744 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
W20260924 05:09:34.559001 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:34.554575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39225 (local address 127.0.98.62:40701)
0924 05:09:34.554695 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:34.554697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:34.554705 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:34.554727 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:34.554727 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:34.554738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:34.554845 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:34.555184 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.555678 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.556352 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.556545 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.556594 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:34.558925 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39225: BlockingRecv error: recv got EOF from 127.0.98.1:39225 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
Sep 24 05:09:34 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:34 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:34.660567 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:34.655781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:42275)
0924 05:09:34.655987 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:34.655999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:34.656069 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:34.656194 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:34.656195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:34.656356 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:34.656403 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:34.656405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.657178 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.657179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:34.657709 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:34.657711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.658029 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.658030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:34.658073 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:34.658548 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:34.658553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:34.660510 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":157,"threads_started":1}
W20260924 05:09:34.660669 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:34.655899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42275 (local address 127.0.98.62:40701)
0924 05:09:34.655978 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:34.655980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:34.656070 (+ 90us) server_negotiation.cc:407] Connection header received
0924 05:09:34.656113 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:34.656114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:34.656125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:34.656178 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:34.656464 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.657097 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.657780 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.657962 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.658011 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:34.660601 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42275: BlockingRecv error: recv got EOF from 127.0.98.2:42275 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 05:09:34 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:34 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:34.786478 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:34.782135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:43321)
0924 05:09:34.782269 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:34.782278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:34.782338 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:34.782484 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:34.782485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:34.782663 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:34.782708 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:34.782710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.783370 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.783371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:34.783884 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:34.783886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.784238 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.784239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:34.784295 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:34.784702 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:34.784706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:34.786409 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
W20260924 05:09:34.786577 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:34.782188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43321 (local address 127.0.98.62:40701)
0924 05:09:34.782234 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:34.782236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:34.782336 (+ 100us) server_negotiation.cc:407] Connection header received
0924 05:09:34.782392 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:34.782393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:34.782407 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:34.782478 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:34.782801 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.783277 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.783971 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:34.784158 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:34.784237 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:34.786512 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43321: BlockingRecv error: recv got EOF from 127.0.98.3:43321 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 05:09:35 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:35 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:35.564023 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:35.559392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:53325)
0924 05:09:35.559523 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:35.559534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:35.559590 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:35.559738 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:35.559739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:35.559896 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:35.559943 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:35.559944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.560566 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.560567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:35.561076 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:35.561077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.561395 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.561396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:35.561432 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:35.561846 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:35.561850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:35.563965 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
W20260924 05:09:35.564121 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:35.559488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53325 (local address 127.0.98.62:40701)
0924 05:09:35.559585 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:35.559587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:35.559626 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:09:35.559647 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:35.559647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:35.559657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:35.559707 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:35.560007 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.560501 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.561149 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.561335 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.561383 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:35.564046 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53325: BlockingRecv error: recv got EOF from 127.0.98.1:53325 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
Sep 24 05:09:35 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:35 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:35.665426 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:35.661068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:50375)
0924 05:09:35.661246 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:35.661257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:35.661316 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:35.661457 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:35.661457 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:35.661619 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:35.661667 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:35.661668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.662286 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.662287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:35.662857 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:35.662858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.663195 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.663196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:35.663235 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:35.663736 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:35.663740 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:35.665368 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
W20260924 05:09:35.665544 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:35.661130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50375 (local address 127.0.98.62:40701)
0924 05:09:35.661200 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:35.661219 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:35.661313 (+ 94us) server_negotiation.cc:407] Connection header received
0924 05:09:35.661383 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:35.661384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:35.661394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:35.661444 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:35.661724 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.662204 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.662949 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.663134 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.663184 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:35.665463 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50375: BlockingRecv error: recv got EOF from 127.0.98.2:50375 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 05:09:35 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:35 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:35.791749 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:35.787014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:33601)
0924 05:09:35.787178 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:35.787191 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:35.787282 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:35.787434 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:35.787435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:35.787598 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:35.787643 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:35.787645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.788320 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.788321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:35.788877 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:35.788879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.789261 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.789262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:35.789323 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:35.789864 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:35.789870 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:35.791693 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
W20260924 05:09:35.791841 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:35.787110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33601 (local address 127.0.98.62:40701)
0924 05:09:35.787182 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:35.787184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:35.787261 (+ 77us) server_negotiation.cc:407] Connection header received
0924 05:09:35.787354 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:35.787355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:35.787368 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:35.787426 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:35.787727 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.788236 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.788959 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:35.789175 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:35.789259 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:35.791775 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33601: BlockingRecv error: recv got EOF from 127.0.98.3:33601 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 05:09:36 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:36 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:36.569180 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:36.564479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:47249)
0924 05:09:36.564640 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:36.564654 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:36.564708 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:36.564851 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:36.564851 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:36.565014 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:36.565060 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:36.565062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.565686 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.565687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:36.566191 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:36.566193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.566528 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.566528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:36.566567 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:36.567043 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:36.567048 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:36.569123 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
W20260924 05:09:36.569299 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:36.564548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47249 (local address 127.0.98.62:40701)
0924 05:09:36.564680 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:36.564682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:36.564693 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:09:36.564765 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:36.564765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:36.564776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:36.564826 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:36.565126 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.565616 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.566263 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.566464 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.566514 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:36.569214 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47249: BlockingRecv error: recv got EOF from 127.0.98.1:47249 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
Sep 24 05:09:36 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:36 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:36.670719 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:36.665887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:51967)
0924 05:09:36.666045 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:36.666056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:36.666149 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:36.666300 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:36.666301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:36.666478 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:36.666535 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:36.666537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.667304 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.667305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:36.667855 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:36.667857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.668310 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.668311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:36.668373 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:36.668780 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:36.668784 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:36.670625 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
W20260924 05:09:36.670780 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:36.665954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51967 (local address 127.0.98.62:40701)
0924 05:09:36.666032 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:36.666034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:36.666128 (+ 94us) server_negotiation.cc:407] Connection header received
0924 05:09:36.666216 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:36.666217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:36.666235 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:36.666293 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:36.666617 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.667220 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.667975 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.668224 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.668308 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:36.670699 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51967: BlockingRecv error: recv got EOF from 127.0.98.2:51967 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 05:09:36 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:36 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:36.796790 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:36.792225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:37939)
0924 05:09:36.792361 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:36.792391 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:36.792455 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:36.792631 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:36.792632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:36.792806 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:36.792851 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:36.792852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.793536 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.793537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:36.794077 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:36.794079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.794452 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.794453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:36.794524 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:36.794988 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:36.794993 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:36.796694 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
W20260924 05:09:36.796831 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:36.792358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37939 (local address 127.0.98.62:40701)
0924 05:09:36.792428 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:36.792431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:36.792441 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:36.792542 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:36.792543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:36.792563 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:36.792625 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:36.792939 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.793440 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.794178 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:36.794368 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:36.794453 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:36.796769 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37939: BlockingRecv error: recv got EOF from 127.0.98.3:37939 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 05:09:37 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:37 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:37.578485 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:37.569659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:35461)
0924 05:09:37.569795 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:37.569805 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:37.569856 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:37.570052 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:37.570053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:37.570205 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:37.570252 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:37.570253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.570956 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.570957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:37.571469 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:37.571470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.571992 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.571994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:37.572065 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:37.576642 (+ 4577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:37.576647 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:37.578426 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260924 05:09:37.578634 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:37.569774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35461 (local address 127.0.98.62:40701)
0924 05:09:37.569920 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:37.569922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:37.569930 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:37.569951 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:37.569952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:37.569961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:37.570031 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:37.570326 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.570891 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.571676 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.571898 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.571990 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:37.578526 (+ 6536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35461: BlockingRecv error: recv got EOF from 127.0.98.1:35461 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
Sep 24 05:09:37 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:37 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:37.675824 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:37.671167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:37931)
0924 05:09:37.671343 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:37.671355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:37.671448 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:37.671633 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:37.671634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:37.671810 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:37.671855 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:37.671857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.672527 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.672528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:37.673054 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:37.673056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.673431 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.673432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:37.673496 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:37.673951 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:37.673955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:37.675750 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
W20260924 05:09:37.675899 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:37.671242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37931 (local address 127.0.98.62:40701)
0924 05:09:37.671331 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:37.671334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:37.671428 (+ 94us) server_negotiation.cc:407] Connection header received
0924 05:09:37.671544 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:37.671545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:37.671557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:37.671629 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:37.671938 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.672437 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.673144 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.673344 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.673429 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:37.675853 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37931: BlockingRecv error: recv got EOF from 127.0.98.2:37931 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 05:09:37 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:37 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:37.802089 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:37.797387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:45991)
0924 05:09:37.797575 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:37.797586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:37.797673 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:37.797844 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:37.797845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:37.798025 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:37.798071 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:37.798073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.798695 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.798696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:37.799292 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:37.799295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.799640 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.799640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:37.799686 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:37.800133 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:37.800137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:37.802032 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
W20260924 05:09:37.802187 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:37.797466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45991 (local address 127.0.98.62:40701)
0924 05:09:37.797547 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:37.797549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:37.797656 (+ 107us) server_negotiation.cc:407] Connection header received
0924 05:09:37.797745 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:37.797746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:37.797761 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:37.797825 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:37.798137 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.798622 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.799390 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:37.799572 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:37.799624 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:37.802117 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45991: BlockingRecv error: recv got EOF from 127.0.98.3:45991 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 05:09:38 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:38 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:38.583541 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:38.579018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:52993)
0924 05:09:38.579150 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:38.579168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:38.579215 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:38.579354 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:38.579354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:38.579513 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:38.579561 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:38.579562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.580178 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.580179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:38.580696 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:38.580697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.581050 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.581051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:38.581103 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:38.581584 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:38.581588 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:38.583482 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
W20260924 05:09:38.583701 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:38.579083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52993 (local address 127.0.98.62:40701)
0924 05:09:38.579208 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:38.579210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:38.579219 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:38.579265 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:38.579265 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:38.579276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:38.579336 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:38.579627 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.580111 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.580773 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.580957 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.581007 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:38.583609 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52993: BlockingRecv error: recv got EOF from 127.0.98.1:52993 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
Sep 24 05:09:38 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:38 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:38.680997 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:38.676343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:51135)
0924 05:09:38.676538 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:38.676549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:38.676619 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:38.676744 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:38.676745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:38.676928 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:38.676973 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:38.676975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.677681 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.677682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:38.678210 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:38.678211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.678588 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.678589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:38.678647 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:38.679151 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:38.679157 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:38.680902 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
W20260924 05:09:38.681052 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:38.676446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51135 (local address 127.0.98.62:40701)
0924 05:09:38.676527 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:38.676530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:38.676600 (+ 70us) server_negotiation.cc:407] Connection header received
0924 05:09:38.676675 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:38.676676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:38.676688 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:38.676740 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:38.677075 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.677590 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.678296 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.678498 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.678587 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:38.680978 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51135: BlockingRecv error: recv got EOF from 127.0.98.2:51135 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 05:09:38 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:38 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:38.807339 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:38.802557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:46891)
0924 05:09:38.802744 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:38.802780 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:38.802857 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:38.803002 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:38.803003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:38.803178 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:38.803224 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:38.803226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.803896 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.803898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:38.804418 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:38.804419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.804800 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.804801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:38.804863 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:38.805368 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:38.805373 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:38.807241 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":118,"threads_started":1}
W20260924 05:09:38.807394 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:38.802642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46891 (local address 127.0.98.62:40701)
0924 05:09:38.802728 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:38.802730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:38.802842 (+ 112us) server_negotiation.cc:407] Connection header received
0924 05:09:38.802931 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:38.802932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:38.802946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:38.802997 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:38.803303 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.803801 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.804501 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:38.804711 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:38.804798 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:38.807326 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46891: BlockingRecv error: recv got EOF from 127.0.98.3:46891 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 05:09:39 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:39 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:39.588500 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:39.584092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:38267)
0924 05:09:39.584228 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:39.584242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:39.584298 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:39.584431 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:39.584432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:39.584591 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:39.584635 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:39.584637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.585245 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.585246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:39.585741 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:39.585742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.586065 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.586066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:39.586100 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:39.586551 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:39.586555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:39.588433 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
W20260924 05:09:39.588609 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:39.584161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38267 (local address 127.0.98.62:40701)
0924 05:09:39.584286 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:39.584287 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:39.584295 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:39.584354 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:39.584354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:39.584365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:39.584412 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:39.584696 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.585182 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.585810 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.586001 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.586052 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:39.588521 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38267: BlockingRecv error: recv got EOF from 127.0.98.1:38267 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
Sep 24 05:09:39 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:39 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:39.687023 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:39.681628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:48961)
0924 05:09:39.681777 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:39.681789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:39.681862 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:39.682009 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:39.682010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:39.682159 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:39.682204 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:39.682206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.683301 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.683302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:39.683828 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:39.683830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.684365 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.684367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:39.684435 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:39.684876 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:39.684881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:39.686954 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
W20260924 05:09:39.687146 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:39.681752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48961 (local address 127.0.98.62:40701)
0924 05:09:39.681815 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:39.681817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:39.681846 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:09:39.681917 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:39.681918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:39.681930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:39.681995 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:39.682279 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.683202 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.683920 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.684277 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.684362 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:39.687076 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48961: BlockingRecv error: recv got EOF from 127.0.98.2:48961 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 05:09:39 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:39 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:39.811970 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:39.807784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:49945)
0924 05:09:39.807942 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:39.807952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:39.808006 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:39.808162 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:39.808163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:39.808313 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:39.808359 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:39.808361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.808970 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.808971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:39.809455 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:39.809457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.809753 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.809754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:39.809790 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:39.810212 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:39.810216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:39.811905 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
W20260924 05:09:39.812093 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:39.807850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49945 (local address 127.0.98.62:40701)
0924 05:09:39.807903 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:39.807904 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:39.807992 (+ 88us) server_negotiation.cc:407] Connection header received
0924 05:09:39.808070 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:39.808071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:39.808084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:39.808144 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:39.808421 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.808907 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.809520 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:39.809692 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:39.809742 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:39.811987 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49945: BlockingRecv error: recv got EOF from 127.0.98.3:49945 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 05:09:40 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:40 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:40.593830 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:40.588977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:49873)
0924 05:09:40.589141 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:40.589153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:40.589221 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:40.589404 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:40.589405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:40.589629 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:40.589700 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:40.589702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.590336 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.590337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:40.590902 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:40.590904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.591239 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.591240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:40.591277 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:40.591725 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:40.591729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:40.593745 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
W20260924 05:09:40.593887 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:40.589063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49873 (local address 127.0.98.62:40701)
0924 05:09:40.589201 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:40.589204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:40.589214 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:40.589307 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:40.589309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:40.589323 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:40.589396 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:40.589776 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.590272 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.590971 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.591176 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.591228 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:40.593835 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49873: BlockingRecv error: recv got EOF from 127.0.98.1:49873 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":45,"threads_started":1}
Sep 24 05:09:40 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:40 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:40.692095 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:40.687475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:45587)
0924 05:09:40.687583 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:40.687608 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:40.687663 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:40.687790 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:40.687791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:40.687947 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:40.687994 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:40.687996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.688662 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.688663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:40.689273 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:40.689275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.689603 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.689604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:40.689640 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:40.690177 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:40.690181 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:40.691994 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
W20260924 05:09:40.692129 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:40.687549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45587 (local address 127.0.98.62:40701)
0924 05:09:40.687638 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:40.687640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:40.687651 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:09:40.687715 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:40.687716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:40.687726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:40.687774 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:40.688065 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.688559 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.689344 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.689526 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.689598 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:40.692070 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45587: BlockingRecv error: recv got EOF from 127.0.98.2:45587 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 05:09:40 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:40 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:40.816910 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:40.812534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.3:44725)
0924 05:09:40.812684 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:40.812701 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:40.812754 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:40.812913 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:40.812914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:40.813074 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:40.813120 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:40.813121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.813715 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.813715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:40.814239 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:40.814240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.814548 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.814549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:40.814586 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:40.815033 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:40.815038 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:40.816853 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
W20260924 05:09:40.817004 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:40.812583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44725 (local address 127.0.98.62:40701)
0924 05:09:40.812731 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:40.812733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:40.812743 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:40.812810 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:40.812811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:40.812822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:40.812881 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:40.813179 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.813655 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.814307 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:40.814488 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:40.814538 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:40.816935 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44725: BlockingRecv error: recv got EOF from 127.0.98.3:44725 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 24 05:09:41 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:41 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:41.598866 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:41.594445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.1:48011)
0924 05:09:41.594596 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:41.594608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:41.594660 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:41.594854 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:41.594855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:41.595009 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:41.595054 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:41.595056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.595691 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.595692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:41.596214 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:41.596216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.596594 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.596595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:41.596651 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:41.597074 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:41.597078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:41.598742 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
W20260924 05:09:41.598958 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:41.594556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48011 (local address 127.0.98.62:40701)
0924 05:09:41.594683 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:41.594684 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:41.594692 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:09:41.594712 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:41.594713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:41.594723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:41.594850 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:41.595120 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.595624 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.596292 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.596491 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.596545 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:41.598886 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48011: BlockingRecv error: recv got EOF from 127.0.98.1:48011 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
Sep 24 05:09:41 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 05:09:41 dist-test-slave-mr2f krb5kdc[8518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 05:09:41.696815 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:41.692615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.98.2:40283)
0924 05:09:41.692736 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:41.692749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:41.692807 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:41.692922 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:41.692923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:41.693079 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:41.693138 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:41.693140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.693766 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.693766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:41.694261 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:41.694262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.694573 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.694574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:41.694610 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:41.695092 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:41.695097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:41.696757 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40701: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
W20260924 05:09:41.696913 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:41.692684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40283 (local address 127.0.98.62:40701)
0924 05:09:41.692744 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:41.692746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:41.692792 (+ 46us) server_negotiation.cc:407] Connection header received
0924 05:09:41.692866 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:41.692867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:41.692878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:41.692918 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:41.693193 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.693700 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.694332 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.694509 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.694562 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:41.696846 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40283: BlockingRecv error: recv got EOF from 127.0.98.2:40283 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:09:41.730866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 9101
I20260924 05:09:41.735016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 9254
I20260924 05:09:41.739233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 9386
I20260924 05:09:41.742918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 9031
2026-09-24T05:09:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17353 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T05:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:41Z Disabled control of system clock
I20260924 05:09:41.770115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40431
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 05:09:41.849047 9582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:41.849251 9582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:41.849268 9582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:41.850445 9582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:41.850484 9582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:41.850497 9582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:41.850510 9582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:41.850564 9582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:41.851867 9582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:41.852067 9582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:41.852346 9582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:41.857314 9588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:41.857347 9587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:41.857393 9582 server_base.cc:1034] running on GCE node
W20260924 05:09:41.857314 9590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:41.857787 9582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:41.858096 9582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:41.859251 9582 hybrid_clock.cc:648] HybridClock initialized: now 1790226581859230 us; error 41 us; skew 500 ppm
I20260924 05:09:41.860396 9582 webserver.cc:460] Webserver started at http://127.0.98.62:35253/ using document root <none> and password file <none>
I20260924 05:09:41.860630 9582 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:41.860698 9582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:41.860849 9582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:41.861864 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "b7184d33b33b493680264a4afe709a77"
format_stamp: "Formatted at 2026-09-24 05:09:41 on dist-test-slave-mr2f"
server_key: "4d4885ae1c52911723d80856eacbc503"
server_key_iv: "b0fc74e0e8cb2d6d448d4a2d2dd477a7"
server_key_version: "encryptionkey@0"
I20260924 05:09:41.862213 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7184d33b33b493680264a4afe709a77"
format_stamp: "Formatted at 2026-09-24 05:09:41 on dist-test-slave-mr2f"
server_key: "4d4885ae1c52911723d80856eacbc503"
server_key_iv: "b0fc74e0e8cb2d6d448d4a2d2dd477a7"
server_key_version: "encryptionkey@0"
I20260924 05:09:41.863590 9582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:41.864342 9596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:41.864538 9582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:09:41.864636 9582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "b7184d33b33b493680264a4afe709a77"
format_stamp: "Formatted at 2026-09-24 05:09:41 on dist-test-slave-mr2f"
server_key: "4d4885ae1c52911723d80856eacbc503"
server_key_iv: "b0fc74e0e8cb2d6d448d4a2d2dd477a7"
server_key_version: "encryptionkey@0"
I20260924 05:09:41.864717 9582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:41.881549 9582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:41.881848 9582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:41.881991 9582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:41.885797 9582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40431
I20260924 05:09:41.885846 9648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40431 every 8 connection(s)
I20260924 05:09:41.886197 9582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:41.886909 9649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:41.889443 9649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77: Bootstrap starting.
I20260924 05:09:41.890033 9649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:41.890329 9649 log.cc:824] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:41.891052 9649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77: No bootstrap required, opened a new log
I20260924 05:09:41.892637 9649 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7184d33b33b493680264a4afe709a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40431 } }
I20260924 05:09:41.892767 9649 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:41.892824 9649 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7184d33b33b493680264a4afe709a77, State: Initialized, Role: FOLLOWER
I20260924 05:09:41.893003 9649 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7184d33b33b493680264a4afe709a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40431 } }
I20260924 05:09:41.893087 9649 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:41.893141 9649 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:41.893188 9649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:41.893723 9649 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7184d33b33b493680264a4afe709a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40431 } }
I20260924 05:09:41.893831 9649 leader_election.cc:302] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7184d33b33b493680264a4afe709a77; no voters:
I20260924 05:09:41.894014 9649 leader_election.cc:288] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:41.894070 9652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:41.894204 9652 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [term 1 LEADER]: Becoming Leader. State: Replica: b7184d33b33b493680264a4afe709a77, State: Running, Role: LEADER
I20260924 05:09:41.894256 9649 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:41.894389 9652 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7184d33b33b493680264a4afe709a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40431 } }
I20260924 05:09:41.894713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 9582
I20260924 05:09:41.894835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:41.894841 9653 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7184d33b33b493680264a4afe709a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7184d33b33b493680264a4afe709a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40431 } } }
I20260924 05:09:41.894912 9653 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:41.895073 392 external_mini_cluster.cc:1428] Setting key 6762af843678bb3d09f2227cc0e1ef29
I20260924 05:09:41.895088 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7184d33b33b493680264a4afe709a77. Latest consensus state: current_term: 1 leader_uuid: "b7184d33b33b493680264a4afe709a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7184d33b33b493680264a4afe709a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40431 } } }
I20260924 05:09:41.895140 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:41.895365 9661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:41.895920 9661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:41.897542 9661 catalog_manager.cc:1347] Generated new cluster ID: 15184eea93e3408d9c84c778a1d0a830
I20260924 05:09:41.897595 9661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:41.900982 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:41.895605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.0.98.62:40431)
0924 05:09:41.896207 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:41.896216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:41.896250 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:09:41.896622 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:41.896630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:41.896707 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:41.896851 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:41.897114 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.898161 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.899079 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:41.899502 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:41.899632 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:41.900331 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:41.900335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:41.900335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:41.900509 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:41.900541 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:41.900551 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:41.900624 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:41.900695 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 05:09:41.900767 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":86,"threads_started":1}
I20260924 05:09:41.915779 9661 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:41.916404 9661 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:41.921662 9661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7184d33b33b493680264a4afe709a77: Generated new TSK 0
I20260924 05:09:41.921823 9661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:09:41.933418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40431
--builtin_ntp_servers=127.0.98.20:39755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 05:09:42.018489 9673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:42.018679 9673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:42.018700 9673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:42.019833 9673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:42.019879 9673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:42.019905 9673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:42.021200 9673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:42.021426 9673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:42.021716 9673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:42.027294 9679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:42.027382 9673 server_base.cc:1034] running on GCE node
W20260924 05:09:42.027307 9678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.027485 9681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:42.027689 9673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:42.028009 9673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:42.029177 9673 hybrid_clock.cc:648] HybridClock initialized: now 1790226582029167 us; error 32 us; skew 500 ppm
I20260924 05:09:42.030375 9673 webserver.cc:460] Webserver started at http://127.0.98.1:39219/ using document root <none> and password file <none>
I20260924 05:09:42.030608 9673 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:42.030676 9673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:42.030838 9673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:42.031824 9673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "5b7548cffd0841bbbec337f1e5dfadbe"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "97bcfa77301af9622b26331c0f149a61"
server_key_iv: "f78609858e857ed5e363efb2ab2bb4b1"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.032191 9673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5b7548cffd0841bbbec337f1e5dfadbe"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "97bcfa77301af9622b26331c0f149a61"
server_key_iv: "f78609858e857ed5e363efb2ab2bb4b1"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.033440 9673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:09:42.034147 9687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.034320 9673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:42.034420 9673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "5b7548cffd0841bbbec337f1e5dfadbe"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "97bcfa77301af9622b26331c0f149a61"
server_key_iv: "f78609858e857ed5e363efb2ab2bb4b1"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.034504 9673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:42.052011 9673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:42.052289 9673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:42.052438 9673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:42.052642 9673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:42.052948 9673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:42.052999 9673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.053031 9673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:42.053057 9673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.058813 9673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38291
I20260924 05:09:42.058866 9800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38291 every 8 connection(s)
I20260924 05:09:42.059250 9673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:09:42.064392 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.059933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53689 (local address 127.0.98.62:40431)
0924 05:09:42.059974 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:42.059976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:42.060221 (+ 245us) server_negotiation.cc:407] Connection header received
0924 05:09:42.060758 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:42.060760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:42.060773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:42.060838 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:42.061496 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.062000 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.063050 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.063255 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.063340 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:42.064100 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:42.064102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:42.064102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:42.064107 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:42.064120 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:42.064124 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:42.064179 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:42.064274 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 05:09:42.064302 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:42.064599 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.059923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40431 (local address 127.0.98.1:53689)
0924 05:09:42.060107 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:42.060138 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:42.060656 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:42.060904 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:42.060908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:42.060920 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:42.061407 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:42.061411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.062097 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.062098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:42.062951 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:42.062954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.063350 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.063351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:42.063484 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:42.063959 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:42.063959 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:42.063985 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:42.064005 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:42.064182 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:42.064187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:42.064189 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:09:42.064275 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 05:09:42.064371 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260924 05:09:42.065047 9801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40431
I20260924 05:09:42.065116 9801 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:42.065299 9801 heartbeater.cc:511] Master 127.0.98.62:40431 requested a full tablet report, sending...
I20260924 05:09:42.065752 9613 ts_manager.cc:194] Registered new tserver with Master: 5b7548cffd0841bbbec337f1e5dfadbe (127.0.98.1:38291)
I20260924 05:09:42.066358 9613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53689
I20260924 05:09:42.068567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 9673
I20260924 05:09:42.068651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:42.068814 392 external_mini_cluster.cc:1428] Setting key bd96d05d1a30d348010c1936253eb04b
I20260924 05:09:42.069585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40431
--builtin_ntp_servers=127.0.98.20:39755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 05:09:42.147709 9804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:42.147899 9804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:42.147920 9804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:42.148998 9804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:42.149029 9804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:42.149052 9804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:42.150219 9804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:42.150383 9804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:42.150596 9804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:09:42.155426 9804 server_base.cc:1034] running on GCE node
W20260924 05:09:42.155404 9809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.155409 9812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.155409 9810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:42.155913 9804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:42.156153 9804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:42.157311 9804 hybrid_clock.cc:648] HybridClock initialized: now 1790226582157300 us; error 42 us; skew 500 ppm
I20260924 05:09:42.158453 9804 webserver.cc:460] Webserver started at http://127.0.98.2:41127/ using document root <none> and password file <none>
I20260924 05:09:42.158687 9804 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:42.158753 9804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:42.158911 9804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:42.159906 9804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "2cc1b450647240bebb3fc6723b01fbec"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "4e27f27eed1e36def29ec2a62f907290"
server_key_iv: "a78b8882ea66d4875799d4bc3ac8a95a"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.160251 9804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2cc1b450647240bebb3fc6723b01fbec"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "4e27f27eed1e36def29ec2a62f907290"
server_key_iv: "a78b8882ea66d4875799d4bc3ac8a95a"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.161489 9804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:09:42.162140 9818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.162307 9804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:42.162395 9804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "2cc1b450647240bebb3fc6723b01fbec"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "4e27f27eed1e36def29ec2a62f907290"
server_key_iv: "a78b8882ea66d4875799d4bc3ac8a95a"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.162474 9804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:42.170542 9804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:42.170832 9804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:42.170969 9804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:42.171167 9804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:42.171449 9804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:42.171500 9804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.171545 9804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:42.171572 9804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.177122 9804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39803
I20260924 05:09:42.177155 9931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39803 every 8 connection(s)
I20260924 05:09:42.177443 9804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:42.182404 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.178140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34837 (local address 127.0.98.62:40431)
0924 05:09:42.178197 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:42.178198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:42.178413 (+ 215us) server_negotiation.cc:407] Connection header received
0924 05:09:42.178975 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:42.178977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:42.178990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:42.179041 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:42.179682 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.180182 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.181146 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.181352 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.181436 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:42.182150 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:42.182151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:42.182152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:42.182157 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:42.182173 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:42.182177 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:42.182225 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:42.182299 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 05:09:42.182327 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:09:42.182614 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.178116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40431 (local address 127.0.98.2:34837)
0924 05:09:42.178308 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:42.178337 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:42.178866 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:42.179104 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:42.179108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:42.179118 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:42.179584 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:42.179589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.180278 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.180279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:42.181048 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:42.181050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.181444 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.181446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:42.181588 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:42.182013 (+ 425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:42.182014 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:42.182045 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:42.182066 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:42.182224 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:42.182229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:42.182232 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:42.182296 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 05:09:42.182400 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260924 05:09:42.183075 9932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40431
I20260924 05:09:42.183148 9932 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:42.183333 9932 heartbeater.cc:511] Master 127.0.98.62:40431 requested a full tablet report, sending...
I20260924 05:09:42.184002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 9804
I20260924 05:09:42.184095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:42.184202 9613 ts_manager.cc:194] Registered new tserver with Master: 2cc1b450647240bebb3fc6723b01fbec (127.0.98.2:39803)
I20260924 05:09:42.184307 392 external_mini_cluster.cc:1428] Setting key 640dd854c7341cf4d8b4e88c05ba58ba
I20260924 05:09:42.184705 9613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34837
I20260924 05:09:42.185505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40431
--builtin_ntp_servers=127.0.98.20:39755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 05:09:42.268306 9935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:42.268524 9935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:42.268559 9935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:42.269668 9935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:42.269731 9935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:42.269774 9935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:42.271102 9935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:42.271343 9935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:42.271611 9935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:42.276458 9941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.276474 9943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.276521 9940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:42.276571 9935 server_base.cc:1034] running on GCE node
I20260924 05:09:42.276858 9935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:42.277115 9935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:42.278258 9935 hybrid_clock.cc:648] HybridClock initialized: now 1790226582278249 us; error 26 us; skew 500 ppm
I20260924 05:09:42.279676 9935 webserver.cc:460] Webserver started at http://127.0.98.3:34655/ using document root <none> and password file <none>
I20260924 05:09:42.279964 9935 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:42.280014 9935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:42.280151 9935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:42.281152 9935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "459d94f3bad240049c1a778f98748529"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "3eae5ca2c8022ff106aa98526b821d16"
server_key_iv: "3ad3b3599c019d16a2fe843a29f495cc"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.281487 9935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "459d94f3bad240049c1a778f98748529"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "3eae5ca2c8022ff106aa98526b821d16"
server_key_iv: "3ad3b3599c019d16a2fe843a29f495cc"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.282684 9935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:42.283416 9949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.283588 9935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:42.283684 9935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "459d94f3bad240049c1a778f98748529"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "3eae5ca2c8022ff106aa98526b821d16"
server_key_iv: "3ad3b3599c019d16a2fe843a29f495cc"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.283759 9935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:42.293305 9935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:42.293567 9935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:42.293708 9935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:42.293915 9935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:42.294256 9935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:42.294332 9935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.294368 9935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:42.294395 9935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.300158 9935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39279
I20260924 05:09:42.300199 10062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39279 every 8 connection(s)
I20260924 05:09:42.300493 9935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:42.305316 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.301110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51857 (local address 127.0.98.62:40431)
0924 05:09:42.301155 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:42.301157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:42.301392 (+ 235us) server_negotiation.cc:407] Connection header received
0924 05:09:42.301833 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:42.301834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:42.301843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:42.301900 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:42.302482 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.303034 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.304141 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.304327 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.304381 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:42.305091 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:42.305092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:42.305093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:42.305097 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:42.305110 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:42.305114 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:42.305156 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:42.305233 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 05:09:42.305257 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:09:42.305505 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.301077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40431 (local address 127.0.98.3:51857)
0924 05:09:42.301260 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:42.301299 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:42.301757 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:42.301949 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:42.301952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:42.301960 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:42.302420 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:42.302424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.303158 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.303159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:42.304061 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:42.304063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:42.304404 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:42.304405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:42.304524 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:42.304951 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:42.304951 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:42.304976 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:42.304996 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:42.305168 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:42.305171 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:42.305173 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:09:42.305222 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:09:42.305294 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260924 05:09:42.305987 10063 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40431
I20260924 05:09:42.306061 10063 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:42.306255 10063 heartbeater.cc:511] Master 127.0.98.62:40431 requested a full tablet report, sending...
I20260924 05:09:42.306608 9612 ts_manager.cc:194] Registered new tserver with Master: 459d94f3bad240049c1a778f98748529 (127.0.98.3:39279)
I20260924 05:09:42.306996 9612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51857
I20260924 05:09:42.310366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 9935
I20260924 05:09:42.310436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:42.310642 392 external_mini_cluster.cc:1428] Setting key 14847688e22805db2c80b27841a8373c
I20260924 05:09:42.311489 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 05:09:42.313091 9666 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:42.312680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.0.98.62:40431)
0924 05:09:42.312711 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:42.312713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:42.312806 (+ 93us) server_negotiation.cc:407] Connection header received
0924 05:09:42.312875 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:42.312876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:42.312885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:42.312922 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:42.313024 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54126: BlockingRecv error: recv got EOF from 127.0.0.1:54126 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:09:42.313364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 9673
I20260924 05:09:42.317838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 9804
I20260924 05:09:42.322002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 9935
I20260924 05:09:42.325937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 9582
2026-09-24T05:09:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (582 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T05:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:42Z Disabled control of system clock
I20260924 05:09:42.351503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36225
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 05:09:42.429996 10074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:42.430197 10074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:42.430218 10074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:42.431371 10074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:42.431411 10074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:42.431425 10074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:42.431438 10074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:42.431450 10074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:42.432703 10074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:42.432884 10074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:42.433120 10074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:42.438000 10079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.437980 10082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:42.438050 10074 server_base.cc:1034] running on GCE node
W20260924 05:09:42.437963 10080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:42.438433 10074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:42.438707 10074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:42.439896 10074 hybrid_clock.cc:648] HybridClock initialized: now 1790226582439863 us; error 51 us; skew 500 ppm
I20260924 05:09:42.441306 10074 webserver.cc:460] Webserver started at http://127.0.98.62:45375/ using document root <none> and password file <none>
I20260924 05:09:42.441545 10074 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:42.441614 10074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:42.441766 10074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:42.442878 10074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "00177a9e8ac64a349b940a935efa8dc3"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "008243706a8ace927046aa74aa3ae2f5"
server_key_iv: "bacdcd7e1c1bd796d5f264a4ab398c58"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.443229 10074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "00177a9e8ac64a349b940a935efa8dc3"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "008243706a8ace927046aa74aa3ae2f5"
server_key_iv: "bacdcd7e1c1bd796d5f264a4ab398c58"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.444590 10074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:09:42.445358 10088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.445523 10074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:42.445618 10074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "00177a9e8ac64a349b940a935efa8dc3"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "008243706a8ace927046aa74aa3ae2f5"
server_key_iv: "bacdcd7e1c1bd796d5f264a4ab398c58"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.445700 10074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:42.448899 10074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:42.449132 10074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:42.449277 10074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:42.452775 10074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36225
I20260924 05:09:42.452805 10140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36225 every 8 connection(s)
I20260924 05:09:42.453148 10074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:42.453770 10141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:42.455212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 10074
I20260924 05:09:42.455315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:42.455508 392 external_mini_cluster.cc:1428] Setting key 2aa8695a40a0e4b85a6c805e8010c8df
I20260924 05:09:42.456925 10141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3: Bootstrap starting.
I20260924 05:09:42.457569 10141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:42.457928 10141 log.cc:824] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:42.458072 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.456022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.0.98.62:36225)
0924 05:09:42.456219 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:42.456227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:42.456259 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:09:42.456757 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:42.456765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:42.456816 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:42.456958 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:42.457591 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:42.457594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:42.457594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:42.457598 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:42.457642 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:42.457655 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:42.457765 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:42.457826 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 05:09:42.457884 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260924 05:09:42.458637 10141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3: No bootstrap required, opened a new log
I20260924 05:09:42.459980 10141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00177a9e8ac64a349b940a935efa8dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36225 } }
I20260924 05:09:42.460100 10141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:42.460121 10141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00177a9e8ac64a349b940a935efa8dc3, State: Initialized, Role: FOLLOWER
I20260924 05:09:42.460232 10141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00177a9e8ac64a349b940a935efa8dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36225 } }
I20260924 05:09:42.460319 10141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:42.460342 10141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:42.460378 10141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:42.461063 10141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00177a9e8ac64a349b940a935efa8dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36225 } }
I20260924 05:09:42.461143 10141 leader_election.cc:302] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00177a9e8ac64a349b940a935efa8dc3; no voters:
I20260924 05:09:42.461251 10141 leader_election.cc:288] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:42.461321 10146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:42.461467 10146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [term 1 LEADER]: Becoming Leader. State: Replica: 00177a9e8ac64a349b940a935efa8dc3, State: Running, Role: LEADER
I20260924 05:09:42.461495 10141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:42.461561 10146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00177a9e8ac64a349b940a935efa8dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36225 } }
I20260924 05:09:42.461937 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00177a9e8ac64a349b940a935efa8dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00177a9e8ac64a349b940a935efa8dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36225 } } }
I20260924 05:09:42.461968 10148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00177a9e8ac64a349b940a935efa8dc3. Latest consensus state: current_term: 1 leader_uuid: "00177a9e8ac64a349b940a935efa8dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00177a9e8ac64a349b940a935efa8dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36225 } } }
I20260924 05:09:42.462056 10148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:42.462041 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:42.462416 10155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:09:42.462921 10155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:42.464289 10155 catalog_manager.cc:1347] Generated new cluster ID: 63852ed70c6f4488b8790d8dce95afd4
I20260924 05:09:42.464339 10155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:42.471369 10155 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:42.471518 10155 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:42.476493 10155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00177a9e8ac64a349b940a935efa8dc3: Generated new TSK 0
I20260924 05:09:42.476651 10155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:09:42.482352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36225
--builtin_ntp_servers=127.0.98.20:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 05:09:42.561235 10165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:42.561476 10165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:42.561520 10165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:42.562870 10165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:42.562912 10165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:42.562939 10165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:42.564764 10165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:42.564981 10165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:42.565258 10165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:42.570806 10173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.570827 10170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.570806 10171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:42.571267 10165 server_base.cc:1034] running on GCE node
I20260924 05:09:42.571487 10165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:42.571712 10165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:42.572861 10165 hybrid_clock.cc:648] HybridClock initialized: now 1790226582572848 us; error 28 us; skew 500 ppm
I20260924 05:09:42.574052 10165 webserver.cc:460] Webserver started at http://127.0.98.1:37019/ using document root <none> and password file <none>
I20260924 05:09:42.574291 10165 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:42.574357 10165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:42.574507 10165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:42.575636 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "52ea9bf75b32453fbc9278a8d7476efd"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "b20992bf274e1d9b3a66fa475171cf8c"
server_key_iv: "6e26c80bce13eb5e109b0d960829543f"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.575996 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "52ea9bf75b32453fbc9278a8d7476efd"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "b20992bf274e1d9b3a66fa475171cf8c"
server_key_iv: "6e26c80bce13eb5e109b0d960829543f"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.577338 10165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:09:42.578096 10179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.578264 10165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:42.578358 10165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "52ea9bf75b32453fbc9278a8d7476efd"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "b20992bf274e1d9b3a66fa475171cf8c"
server_key_iv: "6e26c80bce13eb5e109b0d960829543f"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.578455 10165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:42.581634 10165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:42.581883 10165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:42.582020 10165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:42.582230 10165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:42.582526 10165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:42.582578 10165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.582615 10165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:42.582640 10165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.588150 10165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37385
I20260924 05:09:42.588217 10292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37385 every 8 connection(s)
I20260924 05:09:42.588517 10165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:09:42.595762 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.589190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40871 (local address 127.0.98.62:36225)
0924 05:09:42.589265 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:42.589268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:42.589523 (+ 255us) server_negotiation.cc:407] Connection header received
0924 05:09:42.589916 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:42.589917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:42.589922 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:42.589974 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:42.595561 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:42.595563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:42.595563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:42.595567 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:42.595582 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:42.595586 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:42.595637 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:42.595697 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 05:09:42.595710 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 05:09:42.595990 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.589164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36225 (local address 127.0.98.1:40871)
0924 05:09:42.589354 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:42.589408 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:42.589839 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:42.590034 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:42.590037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:42.590043 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:42.595400 (+ 5357us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:42.595400 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:42.595431 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:42.595461 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:42.595635 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:42.595637 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:42.595640 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:42.595691 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:09:42.595739 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":110,"threads_started":1}
I20260924 05:09:42.596545 10293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36225
I20260924 05:09:42.596635 10293 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:42.596715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 10165
I20260924 05:09:42.596825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:42.596825 10293 heartbeater.cc:511] Master 127.0.98.62:36225 requested a full tablet report, sending...
I20260924 05:09:42.597039 392 external_mini_cluster.cc:1428] Setting key 9823b8950d6437b1104cd06d7b5be5a6
I20260924 05:09:42.597414 10105 ts_manager.cc:194] Registered new tserver with Master: 52ea9bf75b32453fbc9278a8d7476efd (127.0.98.1:37385)
I20260924 05:09:42.598065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36225
--builtin_ntp_servers=127.0.98.20:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 05:09:42.675796 10296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:42.676052 10296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:42.676087 10296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:42.677254 10296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:42.677313 10296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:42.677359 10296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:42.678915 10296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:42.679168 10296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:42.679448 10296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:42.684175 10304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.684212 10302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.684185 10301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:42.684391 10296 server_base.cc:1034] running on GCE node
I20260924 05:09:42.684613 10296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:42.684839 10296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:42.686013 10296 hybrid_clock.cc:648] HybridClock initialized: now 1790226582685994 us; error 33 us; skew 500 ppm
I20260924 05:09:42.687242 10296 webserver.cc:460] Webserver started at http://127.0.98.2:37025/ using document root <none> and password file <none>
I20260924 05:09:42.687479 10296 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:42.687546 10296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:42.687676 10296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:42.688657 10296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "24436fb5609540148e20b5a8a8f6204f"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "6e6d413e9edbc6e3a3e7f4428dc6d87c"
server_key_iv: "9e4a79b87dcda2bfbf07b7cd7db8f148"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.689018 10296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "24436fb5609540148e20b5a8a8f6204f"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "6e6d413e9edbc6e3a3e7f4428dc6d87c"
server_key_iv: "9e4a79b87dcda2bfbf07b7cd7db8f148"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.690234 10296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:09:42.690949 10310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.691123 10296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:42.691224 10296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "24436fb5609540148e20b5a8a8f6204f"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "6e6d413e9edbc6e3a3e7f4428dc6d87c"
server_key_iv: "9e4a79b87dcda2bfbf07b7cd7db8f148"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.691412 10296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:42.694585 10296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:42.694902 10296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:42.695041 10296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:42.695271 10296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:42.695561 10296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:42.695614 10296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.695652 10296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:42.695679 10296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.700824 10296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37709
I20260924 05:09:42.700881 10423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37709 every 8 connection(s)
I20260924 05:09:42.701185 10296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:42.701926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 10296
I20260924 05:09:42.702013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:42.702185 392 external_mini_cluster.cc:1428] Setting key 44476b14b4f1ecc989cdde68a7ecf256
I20260924 05:09:42.703097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36225
--builtin_ntp_servers=127.0.98.20:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 05:09:42.704259 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.701861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42735 (local address 127.0.98.62:36225)
0924 05:09:42.701924 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:42.701926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:42.702128 (+ 202us) server_negotiation.cc:407] Connection header received
0924 05:09:42.702799 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:42.702799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:42.702805 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:42.702886 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:42.703964 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:42.703965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:42.703966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:42.703970 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:42.703987 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:42.703990 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:42.704045 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:42.704157 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 05:09:42.704172 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:09:42.704571 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.701843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36225 (local address 127.0.98.2:42735)
0924 05:09:42.702023 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:42.702060 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:42.702628 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:42.702979 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:42.702982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:42.702990 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:42.703829 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:42.703830 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:42.703853 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:42.703882 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:42.704070 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:42.704076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:42.704079 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:42.704404 (+ 325us) client_negotiation.cc:240] Negotiation successful
0924 05:09:42.704447 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260924 05:09:42.704988 10424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36225
I20260924 05:09:42.705084 10424 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:42.705217 10424 heartbeater.cc:511] Master 127.0.98.62:36225 requested a full tablet report, sending...
I20260924 05:09:42.705545 10104 ts_manager.cc:194] Registered new tserver with Master: 24436fb5609540148e20b5a8a8f6204f (127.0.98.2:37709)
W20260924 05:09:42.784180 10427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:42.784370 10427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:42.784392 10427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:42.785708 10427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:42.785759 10427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:42.785787 10427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:42.787226 10427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:42.787468 10427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:42.787751 10427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:09:42.792616 10427 server_base.cc:1034] running on GCE node
W20260924 05:09:42.792675 10432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.792588 10433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:42.792580 10435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:42.793035 10427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:42.793285 10427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:42.794458 10427 hybrid_clock.cc:648] HybridClock initialized: now 1790226582794440 us; error 28 us; skew 500 ppm
I20260924 05:09:42.795925 10427 webserver.cc:460] Webserver started at http://127.0.98.3:33071/ using document root <none> and password file <none>
I20260924 05:09:42.796171 10427 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:42.796239 10427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:42.796368 10427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:42.797396 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "6b09366c465f447896b367c0c2e9e391"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "0924e2405d052d312f52f61b3717e12f"
server_key_iv: "87824e29c798acf0a431283a5b73a992"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.797751 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6b09366c465f447896b367c0c2e9e391"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "0924e2405d052d312f52f61b3717e12f"
server_key_iv: "87824e29c798acf0a431283a5b73a992"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.799078 10427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:42.799721 10441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.799896 10427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.799980 10427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "6b09366c465f447896b367c0c2e9e391"
format_stamp: "Formatted at 2026-09-24 05:09:42 on dist-test-slave-mr2f"
server_key: "0924e2405d052d312f52f61b3717e12f"
server_key_iv: "87824e29c798acf0a431283a5b73a992"
server_key_version: "encryptionkey@0"
I20260924 05:09:42.800064 10427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:42.803158 10427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:42.803395 10427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:42.803521 10427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:42.803718 10427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:42.803989 10427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:42.804036 10427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.804070 10427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:42.804095 10427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:42.809657 10427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45305
I20260924 05:09:42.809702 10554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45305 every 8 connection(s)
I20260924 05:09:42.809998 10427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:42.812752 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.810678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41925 (local address 127.0.98.62:36225)
0924 05:09:42.810759 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:42.810761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:42.810965 (+ 204us) server_negotiation.cc:407] Connection header received
0924 05:09:42.811405 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:42.811406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:42.811414 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:42.811466 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:42.812510 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:42.812511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:42.812511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:09:42.812517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:42.812531 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:09:42.812536 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:42.812580 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:42.812670 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 05:09:42.812684 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:09:42.812925 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:42.810643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36225 (local address 127.0.98.3:41925)
0924 05:09:42.810862 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:42.810891 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:42.811300 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:42.811537 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:42.811545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:42.811553 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:42.812374 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:09:42.812374 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:42.812399 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:09:42.812422 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:42.812589 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:42.812593 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:42.812596 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:42.812679 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:09:42.812733 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260924 05:09:42.813401 10555 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36225
I20260924 05:09:42.813496 10555 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:42.813629 10555 heartbeater.cc:511] Master 127.0.98.62:36225 requested a full tablet report, sending...
I20260924 05:09:42.813980 10104 ts_manager.cc:194] Registered new tserver with Master: 6b09366c465f447896b367c0c2e9e391 (127.0.98.3:45305)
I20260924 05:09:42.817693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 10427
I20260924 05:09:42.817770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:42.817955 392 external_mini_cluster.cc:1428] Setting key 230ec86a772f071b0578dc311d3dcb05
I20260924 05:09:42.818679 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 05:09:42.820335 10144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:09:42.819918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.0.98.62:36225)
0924 05:09:42.819955 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:42.819957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:42.820095 (+ 138us) server_negotiation.cc:407] Connection header received
0924 05:09:42.820122 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:42.820123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:42.820127 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:42.820170 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:42.820281 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33152: BlockingRecv error: recv got EOF from 127.0.0.1:33152 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:42.820397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 10165
I20260924 05:09:42.824965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 10296
I20260924 05:09:42.828832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 10427
I20260924 05:09:42.832691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 10074
2026-09-24T05:09:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (506 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:42 dist-test-slave-mr2f krb5kdc[10565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:42 dist-test-slave-mr2f krb5kdc[10565](info): set up 2 sockets
Sep 24 05:09:42 dist-test-slave-mr2f krb5kdc[10565](info): commencing operation
krb5kdc: starting...
W20260924 05:09:43.867632 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:43 dist-test-slave-mr2f krb5kdc[10565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:09:43.981396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43283
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:44.058314 10578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:44.058501 10578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:44.058522 10578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:44.059595 10578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:09:44.059630 10578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:44.059644 10578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:44.059657 10578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:09:44.059670 10578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:09:44.060997 10578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:44.061159 10578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:44.061308 10578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:44.066121 10584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:44.066200 10583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:44.066303 10578 server_base.cc:1034] running on GCE node
W20260924 05:09:44.066183 10586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:44.066614 10578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:44.066974 10578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:44.068135 10578 hybrid_clock.cc:648] HybridClock initialized: now 1790226584068120 us; error 44 us; skew 500 ppm
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[10565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:44.070381 10578 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:09:44.070701 10578 webserver.cc:460] Webserver started at http://127.0.98.62:36081/ using document root <none> and password file <none>
I20260924 05:09:44.070950 10578 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:44.070990 10578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:44.071089 10578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:44.071985 10578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "ce058af9a6ba40c2afa1bc22a0c92d27"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "759519e15ca54152a38170fb4ef82ad1"
server_key_iv: "e2503bb0d7b06e5647d23f75c6f860d5"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.072258 10578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce058af9a6ba40c2afa1bc22a0c92d27"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "759519e15ca54152a38170fb4ef82ad1"
server_key_iv: "e2503bb0d7b06e5647d23f75c6f860d5"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.073519 10578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:44.074136 10593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.074309 10578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:44.074396 10578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "ce058af9a6ba40c2afa1bc22a0c92d27"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "759519e15ca54152a38170fb4ef82ad1"
server_key_iv: "e2503bb0d7b06e5647d23f75c6f860d5"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.074460 10578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:44.085421 10578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:44.087704 10578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:44.087864 10578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:44.091548 10578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43283
I20260924 05:09:44.091601 10645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43283 every 8 connection(s)
I20260924 05:09:44.091912 10578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:09:44.092882 10646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:44.095057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 10578
I20260924 05:09:44.095149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:09:44.095212 10646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27: Bootstrap starting.
I20260924 05:09:44.095362 392 external_mini_cluster.cc:1428] Setting key 5fbf33cb768f6b7889ab5ad164d200fb
I20260924 05:09:44.095896 10646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:44.096199 10646 log.cc:824] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:44.096890 10646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27: No bootstrap required, opened a new log
I20260924 05:09:44.099072 10646 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce058af9a6ba40c2afa1bc22a0c92d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260924 05:09:44.099229 10646 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:44.099260 10646 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce058af9a6ba40c2afa1bc22a0c92d27, State: Initialized, Role: FOLLOWER
I20260924 05:09:44.099380 10646 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce058af9a6ba40c2afa1bc22a0c92d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260924 05:09:44.099478 10646 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:09:44.099511 10646 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:09:44.099561 10646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:44.100102 10646 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce058af9a6ba40c2afa1bc22a0c92d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260924 05:09:44.100219 10646 leader_election.cc:302] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce058af9a6ba40c2afa1bc22a0c92d27; no voters:
I20260924 05:09:44.100404 10646 leader_election.cc:288] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:09:44.100456 10651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:44.100638 10651 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [term 1 LEADER]: Becoming Leader. State: Replica: ce058af9a6ba40c2afa1bc22a0c92d27, State: Running, Role: LEADER
I20260924 05:09:44.100710 10646 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:09:44.100771 10651 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce058af9a6ba40c2afa1bc22a0c92d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260924 05:09:44.101214 10651 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce058af9a6ba40c2afa1bc22a0c92d27. Latest consensus state: current_term: 1 leader_uuid: "ce058af9a6ba40c2afa1bc22a0c92d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce058af9a6ba40c2afa1bc22a0c92d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } } }
I20260924 05:09:44.101440 10651 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:44.101370 10652 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce058af9a6ba40c2afa1bc22a0c92d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce058af9a6ba40c2afa1bc22a0c92d27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } } }
I20260924 05:09:44.101861 10652 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27 [sys.catalog]: This master's current role is: LEADER
I20260924 05:09:44.101969 10660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[10565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:44.102411 10660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:09:44.104022 10660 catalog_manager.cc:1347] Generated new cluster ID: 1ca64e485b9046c8a9754b47c13fb953
I20260924 05:09:44.104067 10660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:09:44.106176 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.095861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.0.98.62:43283)
0924 05:09:44.096085 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.096093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.096125 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:09:44.096591 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.096597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.096641 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.096753 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:44.097504 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.098333 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.099450 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.099944 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.100025 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.102975 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:44.102979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:44.102980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:44.102991 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:44.104699 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.105056 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.105059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.105059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.105145 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.105349 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.105350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.105350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.105574 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:44.105772 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.105857 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.105992 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260924 05:09:44.130185 10660 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:09:44.130961 10660 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:09:44.140293 10660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27: Generated new TSK 0
I20260924 05:09:44.140471 10660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:09:44.170194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:44.253397 10672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:44.253648 10672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:44.253688 10672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:44.254985 10672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:44.255041 10672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:44.255093 10672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:09:44.256529 10672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:44.256784 10672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:44.257006 10672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:44.261951 10678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:44.261992 10677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:44.262095 10672 server_base.cc:1034] running on GCE node
W20260924 05:09:44.262107 10680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:44.262408 10672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:44.262637 10672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:44.263797 10672 hybrid_clock.cc:648] HybridClock initialized: now 1790226584263782 us; error 43 us; skew 500 ppm
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[10565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:44.266155 10672 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:09:44.266613 10672 webserver.cc:460] Webserver started at http://127.0.98.1:46301/ using document root <none> and password file <none>
I20260924 05:09:44.266875 10672 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:44.266942 10672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:44.267092 10672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:44.268114 10672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "6635c55aa8e44554a9eadd808cdfcaba"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "30d863666e6a264a77bd6eb78d120d62"
server_key_iv: "1239329066b11ad9351ee7f6e272ccc2"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.268471 10672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6635c55aa8e44554a9eadd808cdfcaba"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "30d863666e6a264a77bd6eb78d120d62"
server_key_iv: "1239329066b11ad9351ee7f6e272ccc2"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.269744 10672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:09:44.270475 10687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.270648 10672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:09:44.270745 10672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "6635c55aa8e44554a9eadd808cdfcaba"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "30d863666e6a264a77bd6eb78d120d62"
server_key_iv: "1239329066b11ad9351ee7f6e272ccc2"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.270834 10672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:44.285225 10672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:44.286993 10672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:44.287151 10672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:44.287371 10672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:44.287667 10672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:44.287722 10672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.287763 10672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:44.287791 10672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.293542 10672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35831
I20260924 05:09:44.293601 10800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35831 every 8 connection(s)
I20260924 05:09:44.293885 10672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:09:44.295184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 10672
I20260924 05:09:44.295261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:09:44.295460 392 external_mini_cluster.cc:1428] Setting key 1af2494c44400c605d97449da7382748
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[10565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:44.303568 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.294533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54327 (local address 127.0.98.62:43283)
0924 05:09:44.294599 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.294601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.294809 (+ 208us) server_negotiation.cc:407] Connection header received
0924 05:09:44.295391 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.295392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.295405 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.295469 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:44.296561 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.297055 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.298057 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.298387 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.298460 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.300591 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:44.300592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:44.300593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:44.300600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:44.302099 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.302571 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.302581 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.302582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.302634 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.302952 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.302953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.302953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.303051 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:44.303105 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.303443 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.303505 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:09:44.303812 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.294505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43283 (local address 127.0.98.1:54327)
0924 05:09:44.294671 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:44.294711 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:44.295215 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:44.295540 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:44.295544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:44.296271 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:44.296466 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.296471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.297149 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.297150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.297939 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.297941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.298473 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.298474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.298579 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.299097 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:44.299104 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:44.300467 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:44.302180 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:44.302187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:44.302190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:44.302491 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:44.302699 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:44.302700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:44.302701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:44.302867 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:44.303123 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:44.303130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:44.303301 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 05:09:44.303411 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 05:09:44.303574 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260924 05:09:44.304339 10801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43283
I20260924 05:09:44.304416 10801 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:44.304551 10801 heartbeater.cc:511] Master 127.0.98.62:43283 requested a full tablet report, sending...
I20260924 05:09:44.305011 10610 ts_manager.cc:194] Registered new tserver with Master: 6635c55aa8e44554a9eadd808cdfcaba (127.0.98.1:35831)
I20260924 05:09:44.305609 10610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:09:44.321038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:44.399991 10806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:44.400193 10806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:44.400215 10806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:44.401434 10806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:44.401539 10806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:44.401594 10806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:09:44.403122 10806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:44.403357 10806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:44.403589 10806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:44.408589 10811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:44.408597 10812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:44.408738 10806 server_base.cc:1034] running on GCE node
W20260924 05:09:44.408603 10814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:44.408993 10806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:44.409268 10806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:44.410425 10806 hybrid_clock.cc:648] HybridClock initialized: now 1790226584410411 us; error 46 us; skew 500 ppm
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[10565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:44.412947 10806 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:09:44.413391 10806 webserver.cc:460] Webserver started at http://127.0.98.2:41021/ using document root <none> and password file <none>
I20260924 05:09:44.413617 10806 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:44.413693 10806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:44.413849 10806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:44.414870 10806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb62dae1f35147ab8ffbae186f8e743f"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "bc9469ac8bf06e9cbbf7446d9c51d1cf"
server_key_iv: "fb771d90b6c98c4b6d50f9f47404cee6"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.415226 10806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb62dae1f35147ab8ffbae186f8e743f"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "bc9469ac8bf06e9cbbf7446d9c51d1cf"
server_key_iv: "fb771d90b6c98c4b6d50f9f47404cee6"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.416486 10806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:09:44.417172 10821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.417343 10806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:44.417446 10806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "cb62dae1f35147ab8ffbae186f8e743f"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "bc9469ac8bf06e9cbbf7446d9c51d1cf"
server_key_iv: "fb771d90b6c98c4b6d50f9f47404cee6"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.417527 10806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:44.432698 10806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:44.435092 10806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:44.435248 10806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:44.435467 10806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:44.435782 10806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:44.435838 10806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.435890 10806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:44.435918 10806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.441455 10806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46581
I20260924 05:09:44.441501 10934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46581 every 8 connection(s)
I20260924 05:09:44.441792 10806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:09:44.445329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 10806
I20260924 05:09:44.445410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:09:44.445607 392 external_mini_cluster.cc:1428] Setting key 96be4386a1da44b691dd6e47b67bfbe5
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[10565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:44.451481 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.442423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41799 (local address 127.0.98.62:43283)
0924 05:09:44.442511 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.442513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.442716 (+ 203us) server_negotiation.cc:407] Connection header received
0924 05:09:44.443307 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.443308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.443339 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.443393 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:44.444500 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.444979 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.446155 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.446354 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.446438 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.448676 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:44.448677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:44.448677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:44.448684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:44.450060 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.450488 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.450490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.450490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.450549 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.450842 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.450843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.450844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.450957 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:44.451037 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.451303 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.451387 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 05:09:44.451683 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.442392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43283 (local address 127.0.98.2:41799)
0924 05:09:44.442597 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:44.442631 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:44.443197 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:44.443468 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:44.443471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:44.444159 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:44.444397 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.444401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.445069 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.445070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.446075 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.446077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.446449 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.446450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.446578 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.447073 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:44.447081 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:44.448560 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:44.450172 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:44.450178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:44.450180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:44.450396 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:44.450621 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:44.450622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:44.450622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:44.450734 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:44.451026 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:44.451032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:44.451169 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 05:09:44.451269 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 05:09:44.451444 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":111,"threads_started":1}
I20260924 05:09:44.452088 10935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43283
I20260924 05:09:44.452172 10935 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:44.452344 10935 heartbeater.cc:511] Master 127.0.98.62:43283 requested a full tablet report, sending...
I20260924 05:09:44.452680 10610 ts_manager.cc:194] Registered new tserver with Master: cb62dae1f35147ab8ffbae186f8e743f (127.0.98.2:46581)
I20260924 05:09:44.453058 10610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41799
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:09:44.470324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:09:44.549713 10940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:09:44.549896 10940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:09:44.549918 10940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:09:44.551142 10940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:09:44.551182 10940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:09:44.551209 10940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:09:44.552668 10940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:09:44.552874 10940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:09:44.553045 10940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:09:44.557823 10946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:09:44.557880 10948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:44.557901 10940 server_base.cc:1034] running on GCE node
W20260924 05:09:44.557833 10945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:09:44.558283 10940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:09:44.558521 10940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:09:44.559656 10940 hybrid_clock.cc:648] HybridClock initialized: now 1790226584559647 us; error 27 us; skew 500 ppm
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[10565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:09:44.562098 10940 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:09:44.562582 10940 webserver.cc:460] Webserver started at http://127.0.98.3:45077/ using document root <none> and password file <none>
I20260924 05:09:44.562875 10940 fs_manager.cc:359] Metadata directory not provided
I20260924 05:09:44.562949 10940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:09:44.563105 10940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:09:44.564117 10940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "ce72045932874ed9a0c0f5a97be68aca"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "cedb0fba337ef171da34e5997fb2cd32"
server_key_iv: "2bdfc2390bcf6601a9de422ba31687ba"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.564502 10940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ce72045932874ed9a0c0f5a97be68aca"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "cedb0fba337ef171da34e5997fb2cd32"
server_key_iv: "2bdfc2390bcf6601a9de422ba31687ba"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.565763 10940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:09:44.566430 10955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.566606 10940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:09:44.566704 10940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "ce72045932874ed9a0c0f5a97be68aca"
format_stamp: "Formatted at 2026-09-24 05:09:44 on dist-test-slave-mr2f"
server_key: "cedb0fba337ef171da34e5997fb2cd32"
server_key_iv: "2bdfc2390bcf6601a9de422ba31687ba"
server_key_version: "encryptionkey@0"
I20260924 05:09:44.566809 10940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:09:44.585296 10940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:09:44.587106 10940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:09:44.587272 10940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:09:44.587507 10940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:09:44.587863 10940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:09:44.587926 10940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.587968 10940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:09:44.587993 10940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:09:44.593850 10940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36759
I20260924 05:09:44.593909 11068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36759 every 8 connection(s)
I20260924 05:09:44.594182 10940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:09:44.594669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 10940
I20260924 05:09:44.594802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:09:44.595044 392 external_mini_cluster.cc:1428] Setting key e4f125901954db5bf01ecfb35598e718
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[10565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:09:44.603502 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.594879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46885 (local address 127.0.98.62:43283)
0924 05:09:44.594940 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.594942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.595182 (+ 240us) server_negotiation.cc:407] Connection header received
0924 05:09:44.595849 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.595850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.595873 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.595924 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:44.596898 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.597388 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.598428 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.598606 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.598656 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.600772 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:44.600774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:44.600774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:44.600780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:44.602205 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.602588 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.602588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.602589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.602638 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.602906 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.602906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.602907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.603034 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:44.603110 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.603374 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.603438 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:09:44.603646 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.594851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43283 (local address 127.0.98.3:46885)
0924 05:09:44.595040 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:44.595087 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:44.595764 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:44.595995 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:44.595999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:44.596546 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:09:44.596806 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.596812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.597497 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.597498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.598351 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.598353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.598678 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.598679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.598838 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.599318 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:09:44.599326 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:09:44.600693 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:09:44.602284 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:44.602289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:44.602292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:44.602523 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:44.602697 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:09:44.602697 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:09:44.602698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:09:44.602847 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:09:44.603123 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:09:44.603129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:09:44.603262 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 05:09:44.603338 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 05:09:44.603447 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260924 05:09:44.604152 11069 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43283
I20260924 05:09:44.604246 11069 heartbeater.cc:464] Registering TS with master...
I20260924 05:09:44.604434 11069 heartbeater.cc:511] Master 127.0.98.62:43283 requested a full tablet report, sending...
I20260924 05:09:44.604771 10610 ts_manager.cc:194] Registered new tserver with Master: ce72045932874ed9a0c0f5a97be68aca (127.0.98.3:36759)
I20260924 05:09:44.605116 10610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46885
I20260924 05:09:44.606737 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:09:44.613355 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.607877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.0.98.62:43283)
0924 05:09:44.607943 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.607945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.608079 (+ 134us) server_negotiation.cc:407] Connection header received
0924 05:09:44.608135 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.608136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.608143 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.608188 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:09:44.608822 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.609283 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.609977 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.610148 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.610204 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.611324 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:09:44.611325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:09:44.611326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:09:44.611331 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:09:44.612414 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.612706 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.612707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.612707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.612752 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:09:44.612905 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:09:44.612905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:09:44.612906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:09:44.613050 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:09:44.613124 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.613208 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.613260 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:09:44.614706 10610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:09:44.616297 10610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:09:44.622749 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.618266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35831 (local address 127.0.0.1:34064)
0924 05:09:44.618556 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:44.618598 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:44.618663 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:44.618999 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:44.619002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:44.619003 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:44.619066 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.619068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.620188 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.620189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.621448 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.621450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.622424 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.622425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.622511 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.622525 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:09:44.622600 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:09:44.622658 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":26,"thread_start_us":67,"threads_started":1}
I20260924 05:09:44.622752 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.618470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.0.98.3:36759)
0924 05:09:44.618615 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.618621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.618957 (+ 336us) server_negotiation.cc:407] Connection header received
0924 05:09:44.619075 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.619077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.619105 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.619173 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:44.619361 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.620411 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.621606 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.622259 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.622331 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.622386 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.622633 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.622673 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260924 05:09:44.623046 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.618499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34064 (local address 127.0.98.1:35831)
0924 05:09:44.618647 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.618651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.618661 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:09:44.618795 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.618797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.618814 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.618904 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:44.619147 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.620092 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.621520 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.622259 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.622331 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.622386 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.622910 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.622952 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260924 05:09:44.623278 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.618268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46581 (local address 127.0.0.1:43192)
0924 05:09:44.618803 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:44.618813 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:44.618859 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:44.619096 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:44.619097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:44.619099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:44.619153 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.619154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.621624 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.621624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.622239 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.622240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.623034 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.623035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.623101 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.623106 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:09:44.623168 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 05:09:44.623212 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":51,"threads_started":1}
I20260924 05:09:44.623298 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.618358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.0.98.2:46581)
0924 05:09:44.618543 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.618548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.618857 (+ 309us) server_negotiation.cc:407] Connection header received
0924 05:09:44.618931 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.618934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.618950 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.619018 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:44.619241 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.620411 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.622323 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.622951 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.623028 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.623080 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.623181 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.623225 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260924 05:09:44.622748 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.618285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36759 (local address 127.0.0.1:55026)
0924 05:09:44.618842 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:44.618852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:44.618987 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:44.619245 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:44.619246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:44.619248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:44.619281 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.619282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.620520 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.620521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.621535 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.621536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.622429 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.622429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.622511 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.622526 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 05:09:44.622600 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 05:09:44.622658 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":6,"thread_start_us":47,"threads_started":1}
I20260924 05:09:44.623787 10735 tablet_service.cc:1467] Processing CreateTablet for tablet 11da76ebeb714df3b1cf25cc151a1868 (DEFAULT_TABLE table=test-table [id=c6b5af1f1bb54efabf9df66d5f0990ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:44.624151 10735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da76ebeb714df3b1cf25cc151a1868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:44.624119 10869 tablet_service.cc:1467] Processing CreateTablet for tablet 11da76ebeb714df3b1cf25cc151a1868 (DEFAULT_TABLE table=test-table [id=c6b5af1f1bb54efabf9df66d5f0990ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:44.624351 10869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da76ebeb714df3b1cf25cc151a1868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:44.624366 11003 tablet_service.cc:1467] Processing CreateTablet for tablet 11da76ebeb714df3b1cf25cc151a1868 (DEFAULT_TABLE table=test-table [id=c6b5af1f1bb54efabf9df66d5f0990ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:09:44.624644 11003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11da76ebeb714df3b1cf25cc151a1868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:09:44.626901 11083 tablet_bootstrap.cc:492] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: Bootstrap starting.
I20260924 05:09:44.627451 11084 tablet_bootstrap.cc:492] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba: Bootstrap starting.
I20260924 05:09:44.627596 11083 tablet_bootstrap.cc:654] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:44.627771 11085 tablet_bootstrap.cc:492] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: Bootstrap starting.
I20260924 05:09:44.628192 11084 tablet_bootstrap.cc:654] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:44.628355 11085 tablet_bootstrap.cc:654] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: Neither blocks nor log segments found. Creating new log.
I20260924 05:09:44.628408 11083 log.cc:824] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:44.628734 11084 log.cc:824] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:44.628952 11085 log.cc:824] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: Log is configured to *not* fsync() on all Append() calls
I20260924 05:09:44.629313 11083 tablet_bootstrap.cc:492] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: No bootstrap required, opened a new log
I20260924 05:09:44.629462 11083 ts_tablet_manager.cc:1397] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:09:44.629520 11084 tablet_bootstrap.cc:492] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba: No bootstrap required, opened a new log
I20260924 05:09:44.629576 11084 ts_tablet_manager.cc:1397] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:09:44.629683 11085 tablet_bootstrap.cc:492] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: No bootstrap required, opened a new log
I20260924 05:09:44.629778 11085 ts_tablet_manager.cc:1397] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:09:44.631170 11084 raft_consensus.cc:348] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } }
I20260924 05:09:44.631330 11084 raft_consensus.cc:374] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:44.631389 11084 raft_consensus.cc:729] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6635c55aa8e44554a9eadd808cdfcaba, State: Initialized, Role: FOLLOWER
I20260924 05:09:44.631512 11084 consensus_queue.cc:260] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } }
I20260924 05:09:44.631623 11083 raft_consensus.cc:348] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } }
I20260924 05:09:44.631644 11085 raft_consensus.cc:348] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } }
I20260924 05:09:44.631803 11083 raft_consensus.cc:374] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:44.631803 11085 raft_consensus.cc:374] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:09:44.631788 11084 ts_tablet_manager.cc:1428] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:09:44.631837 11083 raft_consensus.cc:729] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb62dae1f35147ab8ffbae186f8e743f, State: Initialized, Role: FOLLOWER
I20260924 05:09:44.631837 11085 raft_consensus.cc:729] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce72045932874ed9a0c0f5a97be68aca, State: Initialized, Role: FOLLOWER
I20260924 05:09:44.631927 10801 heartbeater.cc:503] Master 127.0.98.62:43283 was elected leader, sending a full tablet report...
I20260924 05:09:44.631985 11083 consensus_queue.cc:260] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } }
I20260924 05:09:44.631990 11085 consensus_queue.cc:260] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } }
I20260924 05:09:44.632241 11085 ts_tablet_manager.cc:1428] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:09:44.632376 11069 heartbeater.cc:503] Master 127.0.98.62:43283 was elected leader, sending a full tablet report...
I20260924 05:09:44.632241 11083 ts_tablet_manager.cc:1428] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 05:09:44.632475 10935 heartbeater.cc:503] Master 127.0.98.62:43283 was elected leader, sending a full tablet report...
W20260924 05:09:44.692381 10936 tablet.cc:2369] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:09:44.726553 11090 raft_consensus.cc:482] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:09:44.726715 11090 raft_consensus.cc:504] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } }
I20260924 05:09:44.727142 11090 leader_election.cc:288] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6635c55aa8e44554a9eadd808cdfcaba (127.0.98.1:35831), cb62dae1f35147ab8ffbae186f8e743f (127.0.98.2:46581)
I20260924 05:09:44.729889 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.727402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52273 (local address 127.0.98.1:35831)
0924 05:09:44.727449 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.727452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.727479 (+ 27us) server_negotiation.cc:407] Connection header received
0924 05:09:44.727509 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.727510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.727523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.727567 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:44.727752 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.728224 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.729155 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.729525 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.729616 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.729627 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.729788 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.729822 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:09:44.729889 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.727323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35831 (local address 127.0.98.3:52273)
0924 05:09:44.727367 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:44.727380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:44.727430 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:44.727608 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:44.727609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:44.727611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:44.727673 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.727675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.728320 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.728321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.729054 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.729056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.729623 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.729624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.729708 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.729711 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:09:44.729781 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:09:44.729821 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 05:09:44.730166 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.727470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59113 (local address 127.0.98.2:46581)
0924 05:09:44.727512 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.727518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.727664 (+ 146us) server_negotiation.cc:407] Connection header received
0924 05:09:44.727726 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.727727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.727739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.727807 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:09:44.727966 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.728427 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.729375 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.729761 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.729844 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.729853 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.730055 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.730105 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:09:44.730278 10755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da76ebeb714df3b1cf25cc151a1868" candidate_uuid: "ce72045932874ed9a0c0f5a97be68aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6635c55aa8e44554a9eadd808cdfcaba" is_pre_election: true
I20260924 05:09:44.730363 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.727407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46581 (local address 127.0.98.3:59113)
0924 05:09:44.727595 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:44.727603 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:09:44.727669 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:09:44.727823 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:09:44.727824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:09:44.727825 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:09:44.727898 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.727899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.728518 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.728519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.729275 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:09:44.729277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.729859 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.729860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:09:44.729955 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.729960 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:09:44.730245 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 05:09:44.730290 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":115,"threads_started":1}
I20260924 05:09:44.730434 10755 raft_consensus.cc:2454] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce72045932874ed9a0c0f5a97be68aca in term 0.
I20260924 05:09:44.730684 10956 leader_election.cc:302] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6635c55aa8e44554a9eadd808cdfcaba, ce72045932874ed9a0c0f5a97be68aca; no voters:
I20260924 05:09:44.730724 10889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da76ebeb714df3b1cf25cc151a1868" candidate_uuid: "ce72045932874ed9a0c0f5a97be68aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb62dae1f35147ab8ffbae186f8e743f" is_pre_election: true
I20260924 05:09:44.730824 11090 raft_consensus.cc:2781] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:09:44.730877 11090 raft_consensus.cc:482] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:09:44.730911 11090 raft_consensus.cc:3037] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:44.730902 10889 raft_consensus.cc:2454] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce72045932874ed9a0c0f5a97be68aca in term 0.
I20260924 05:09:44.731647 11090 raft_consensus.cc:504] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } }
I20260924 05:09:44.731794 11090 leader_election.cc:288] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [CANDIDATE]: Term 1 election: Requested vote from peers 6635c55aa8e44554a9eadd808cdfcaba (127.0.98.1:35831), cb62dae1f35147ab8ffbae186f8e743f (127.0.98.2:46581)
I20260924 05:09:44.731931 10755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da76ebeb714df3b1cf25cc151a1868" candidate_uuid: "ce72045932874ed9a0c0f5a97be68aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6635c55aa8e44554a9eadd808cdfcaba"
I20260924 05:09:44.731953 10889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11da76ebeb714df3b1cf25cc151a1868" candidate_uuid: "ce72045932874ed9a0c0f5a97be68aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb62dae1f35147ab8ffbae186f8e743f"
I20260924 05:09:44.732014 10889 raft_consensus.cc:3037] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:44.732008 10755 raft_consensus.cc:3037] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:09:44.732839 10889 raft_consensus.cc:2454] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce72045932874ed9a0c0f5a97be68aca in term 1.
I20260924 05:09:44.732839 10755 raft_consensus.cc:2454] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce72045932874ed9a0c0f5a97be68aca in term 1.
I20260924 05:09:44.733000 10956 leader_election.cc:302] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6635c55aa8e44554a9eadd808cdfcaba, ce72045932874ed9a0c0f5a97be68aca; no voters:
I20260924 05:09:44.733080 11090 raft_consensus.cc:2781] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:09:44.733224 11090 raft_consensus.cc:686] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 1 LEADER]: Becoming Leader. State: Replica: ce72045932874ed9a0c0f5a97be68aca, State: Running, Role: LEADER
I20260924 05:09:44.733314 11090 consensus_queue.cc:237] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } }
I20260924 05:09:44.734064 10609 catalog_manager.cc:5495] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca reported cstate change: term changed from 0 to 1, leader changed from <none> to ce72045932874ed9a0c0f5a97be68aca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ce72045932874ed9a0c0f5a97be68aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce72045932874ed9a0c0f5a97be68aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36759 } health_report { overall_health: HEALTHY } } }
I20260924 05:09:44.765686 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.763298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.0.98.3:36759)
0924 05:09:44.763338 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:09:44.763341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:09:44.763350 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:09:44.763425 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:09:44.763426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:09:44.763449 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:09:44.763524 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:09:44.763673 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.764128 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.764880 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:09:44.765085 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:09:44.765165 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:09:44.765347 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:09:44.765440 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:09:44.765535 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 05:09:44.765555 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 05:09:44.765575 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:09:44.767436 10755 raft_consensus.cc:1261] T 11da76ebeb714df3b1cf25cc151a1868 P 6635c55aa8e44554a9eadd808cdfcaba [term 1 FOLLOWER]: Refusing update from remote peer ce72045932874ed9a0c0f5a97be68aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:44.767431 10889 raft_consensus.cc:1261] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [term 1 FOLLOWER]: Refusing update from remote peer ce72045932874ed9a0c0f5a97be68aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:09:44.767815 11090 consensus_queue.cc:1035] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb62dae1f35147ab8ffbae186f8e743f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:44.767944 11093 consensus_queue.cc:1035] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "6635c55aa8e44554a9eadd808cdfcaba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:09:44.770133 11096 mvcc.cc:205] Tried to move back new op lower bound from 7332768091204517888 to 7332768091068268544. Current Snapshot: MvccSnapshot[applied={T|T < 7332768091204517888}]
I20260924 05:09:44.770363 11097 mvcc.cc:205] Tried to move back new op lower bound from 7332768091204517888 to 7332768091068268544. Current Snapshot: MvccSnapshot[applied={T|T < 7332768091204517888}]
I20260924 05:09:44.771183 11098 mvcc.cc:205] Tried to move back new op lower bound from 7332768091204517888 to 7332768091068268544. Current Snapshot: MvccSnapshot[applied={T|T < 7332768091204517888 or (T in {7332768091204517888})}]
I20260924 05:09:44.772646 10609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:44.772734 10609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:09:44.773458 10609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce058af9a6ba40c2afa1bc22a0c92d27: Sending DeleteTablet for 3 replicas of tablet 11da76ebeb714df3b1cf25cc151a1868
I20260924 05:09:44.773784 10869 tablet_service.cc:1514] Processing DeleteTablet for tablet 11da76ebeb714df3b1cf25cc151a1868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43192
I20260924 05:09:44.773802 10735 tablet_service.cc:1514] Processing DeleteTablet for tablet 11da76ebeb714df3b1cf25cc151a1868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34064
I20260924 05:09:44.773788 11003 tablet_service.cc:1514] Processing DeleteTablet for tablet 11da76ebeb714df3b1cf25cc151a1868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:09:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55026
I20260924 05:09:44.774034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 10672
I20260924 05:09:44.774132 11107 tablet_replica.cc:330] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: stopping tablet replica
I20260924 05:09:44.774212 11107 raft_consensus.cc:2229] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:09:44.774299 11107 raft_consensus.cc:2258] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:44.774410 11108 tablet_replica.cc:330] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: stopping tablet replica
I20260924 05:09:44.774528 11108 raft_consensus.cc:2229] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:09:44.774600 11108 raft_consensus.cc:2258] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:09:44.775480 11107 ts_tablet_manager.cc:1905] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:09:44.777078 11107 ts_tablet_manager.cc:1918] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:44.777172 11107 log.cc:1196] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/11da76ebeb714df3b1cf25cc151a1868
I20260924 05:09:44.777457 11107 ts_tablet_manager.cc:1939] T 11da76ebeb714df3b1cf25cc151a1868 P cb62dae1f35147ab8ffbae186f8e743f: Deleting consensus metadata
I20260924 05:09:44.777851 10595 catalog_manager.cc:4841] TS cb62dae1f35147ab8ffbae186f8e743f (127.0.98.2:46581): tablet 11da76ebeb714df3b1cf25cc151a1868 (table test-table [id=c6b5af1f1bb54efabf9df66d5f0990ff]) successfully deleted
I20260924 05:09:44.779048 11108 ts_tablet_manager.cc:1905] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:09:44.779786 10594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35831 (error 108)
I20260924 05:09:44.780022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 10806
I20260924 05:09:44.780674 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:09:44.780118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35831 (local address 127.0.0.1:34066)
0924 05:09:44.780564 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 05:09:44.780622 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423}
I20260924 05:09:44.781297 11108 ts_tablet_manager.cc:1918] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:09:44.781344 11108 log.cc:1196] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/11da76ebeb714df3b1cf25cc151a1868
I20260924 05:09:44.781584 11108 ts_tablet_manager.cc:1939] T 11da76ebeb714df3b1cf25cc151a1868 P ce72045932874ed9a0c0f5a97be68aca: Deleting consensus metadata
I20260924 05:09:44.781967 10596 catalog_manager.cc:4841] TS ce72045932874ed9a0c0f5a97be68aca (127.0.98.3:36759): tablet 11da76ebeb714df3b1cf25cc151a1868 (table test-table [id=c6b5af1f1bb54efabf9df66d5f0990ff]) successfully deleted
W20260924 05:09:44.785624 10594 catalog_manager.cc:4584] TS 6635c55aa8e44554a9eadd808cdfcaba (127.0.98.1:35831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11da76ebeb714df3b1cf25cc151a1868: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35831: connect: Connection refused (error 111)
I20260924 05:09:44.785899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 10940
I20260924 05:09:44.790371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 10578
2026-09-24T05:09:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1959 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[11112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[11112](info): set up 2 sockets
Sep 24 05:09:44 dist-test-slave-mr2f krb5kdc[11112](info): commencing operation
krb5kdc: starting...
W20260924 05:09:45.811810 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:09:45 dist-test-slave-mr2f krb5kdc[11112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:09:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 05:09:45.975332 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhdUPbm/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/postgres -l logfile start
2026-09-24 05:09:48.431 UTC [11142] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 05:09:48.431 UTC [11142] LOG: listening on IPv4 address "127.0.98.20", port 33539
2026-09-24 05:09:48.434 UTC [11142] LOG: listening on Unix socket "/tmp/.s.PGSQL.33539"
2026-09-24 05:09:48.456 UTC [11145] LOG: database system was shut down at 2026-09-24 05:09:47 UTC
2026-09-24 05:09:48.460 UTC [11142] LOG: database system is ready to accept connections
I20260924 05:09:49.407686 392 mini_postgres.cc:96] Postgres bound to 33539
2026-09-24 05:09:49.413 UTC [11153] FATAL: database "slave" does not exist
127.0.98.20:33539 - accepting connections
I20260924 05:09:49.413893 392 mini_ranger.cc:161] Starting Ranger...
I20260924 05:09:49.427783 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 05:09:49.913390 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 05:09:49.913489 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-admin
2026-09-24 05:09:50,180 [I] DB FLAVOR :POSTGRES
2026-09-24 05:09:50,181 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:09:50,181 [I] Checking connection..
2026-09-24 05:09:50,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 05:09:50,423 [I] Checking connection passed.
2026-09-24 05:09:50,423 [I] --------- Verifying version history table ---------
2026-09-24 05:09:50,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:09:50,649 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:09:50,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:09:50,894 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:09:50,895 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 05:09:50,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 05:09:51.123 UTC [11217] WARNING: there is no transaction in progress
2026-09-24 05:09:51,129 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 05:09:51,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:09:51,369 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 05:09:51,369 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:09:51,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 05:09:51,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 05:09:51,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mr2f.c.gcp-upstream.internal','N') ;"
2026-09-24 05:09:52,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:09:52,317 [I] Table x_portal_user does not exist in database ranger
2026-09-24 05:09:52,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:09:52,570 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 05:09:52,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:09:52,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:09:53,055 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 05:09:53,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 05:09:53.996 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.013 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.028 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.043 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.059 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.075 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.128 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.135 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.148 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.158 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.168 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.182 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.191 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.197 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.203 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54.210 UTC [11325] WARNING: there is no transaction in progress
2026-09-24 05:09:54,814 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 05:09:54,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 05:09:55,058 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 05:09:55,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:09:55,305 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 05:09:55,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:09:55,556 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 05:09:55,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:09:55,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:09:56,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mr2f.c.gcp-upstream.internal';"
2026-09-24 05:09:56,262 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 05:09:56,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 05:09:56,517 [I] DB_PATCHES have already been applied
I20260924 05:09:56.523169 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhdUPbm/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 05:09:56.523228 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 05:09:56.525977 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:42191
Sep 24, 2026 5:09:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:09:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:09:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 5:09:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 5:09:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 5:09:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 5:09:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 05:09:57.604560 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.191s user 0.000s sys 0.008s
Sep 24, 2026 5:10:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:10:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 5:10:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:10:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:10:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 05:10:17.885360 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 05:10:17 dist-test-slave-mr2f krb5kdc[11112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226617, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 05:10:17.964766 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 05:10:17.973588 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 05:10:18.779470 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 05:10:18.779557 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-kms
I20260924 05:10:18.779578 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:33539
2026-09-24 05:10:18,989 [I] DB FLAVOR :POSTGRES
2026-09-24 05:10:18,989 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:10:18,990 [I] Checking connection
2026-09-24 05:10:18,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 05:10:19,257 [I] connection success
2026-09-24 05:10:19,257 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 05:10:19,257 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 05:10:19,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 05:10:19,545 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 05:10:19,545 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:10:19,546 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 05:10:19,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 05:10:19,872 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 05:10:20.299223 392 mini_ranger_kms.cc:310] Created kms service
I20260924 05:10:20.504751 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 05:10:20.546602 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 05:10:20 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226617, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:10:20.899255 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 05:10:20.899370 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 05:10:20.899392 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 05:10:20.902907 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11505
I20260924 05:10:20.903008 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:50011
Sep 24, 2026 5:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:10:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:10:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:10:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:10:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 05:10:23.107066 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.142s user 0.000s sys 0.191s
I20260924 05:10:23.107179 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 05:10:23.107259 392 mini_ranger_kms.cc:401] 127.0.98.20:50011/kms/v1/keys
Sep 24 05:10:23 dist-test-slave-mr2f krb5kdc[11112](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226623, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 05:10:23 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226623, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:10:24 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226623, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:10:26 dist-test-slave-mr2f krb5kdc[11112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:10:26.225760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37865
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37865
--ranger_config_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50011/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:10:26.310523 11556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:10:26.310737 11556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:10:26.310787 11556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:10:26.311866 11556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:10:26.311902 11556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:10:26.311916 11556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:10:26.311929 11556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:10:26.311941 11556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:10:26.313370 11556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:50011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37865
--ranger_config_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:10:26.313540 11556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:10:26.313689 11556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:10:26.318292 11562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:10:26.318312 11564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:10:26.318311 11561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:10:26.318457 11556 server_base.cc:1034] running on GCE node
I20260924 05:10:26.318823 11556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:10:26.319144 11556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:10:26.320302 11556 hybrid_clock.cc:648] HybridClock initialized: now 1790226626320292 us; error 38 us; skew 500 ppm
Sep 24 05:10:26 dist-test-slave-mr2f krb5kdc[11112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:10:26.322734 11556 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:10:26.323230 11556 webserver.cc:460] Webserver started at http://127.0.98.62:41515/ using document root <none> and password file <none>
I20260924 05:10:26.323441 11556 fs_manager.cc:359] Metadata directory not provided
I20260924 05:10:26.323495 11556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:10:26.323632 11556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:10:26 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:10:26.376969 11556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "a4404b88dd534a11bae5a17c9a912074"
format_stamp: "Formatted at 2026-09-24 05:10:26 on dist-test-slave-mr2f"
server_key: "e568fca7992a2d35de53fa7ba0494892"
server_key_iv: "da183267949c9c8ee745982ac865286b"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:26.377498 11556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4404b88dd534a11bae5a17c9a912074"
format_stamp: "Formatted at 2026-09-24 05:10:26 on dist-test-slave-mr2f"
server_key: "e568fca7992a2d35de53fa7ba0494892"
server_key_iv: "da183267949c9c8ee745982ac865286b"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:26.379061 11556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:10:26.395020 11571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:26.395262 11556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:10:26.395377 11556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "a4404b88dd534a11bae5a17c9a912074"
format_stamp: "Formatted at 2026-09-24 05:10:26 on dist-test-slave-mr2f"
server_key: "e568fca7992a2d35de53fa7ba0494892"
server_key_iv: "da183267949c9c8ee745982ac865286b"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:26.395483 11556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:10:26.420002 11556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:10:26.422102 11556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:10:26.422217 11556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:10:26.426170 11556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37865
I20260924 05:10:26.426230 11623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37865 every 8 connection(s)
I20260924 05:10:26.426568 11556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:10:26.427227 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:10:26.429840 11624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074: Bootstrap starting.
I20260924 05:10:26.430363 11624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074: Neither blocks nor log segments found. Creating new log.
I20260924 05:10:26.430581 11624 log.cc:824] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074: Log is configured to *not* fsync() on all Append() calls
I20260924 05:10:26.431138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 11556
I20260924 05:10:26.431262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:10:26.431324 11624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074: No bootstrap required, opened a new log
I20260924 05:10:26.431466 392 external_mini_cluster.cc:1428] Setting key cf42d68db300071ff479d0518a6362b8
I20260924 05:10:26.433748 11624 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4404b88dd534a11bae5a17c9a912074" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37865 } }
I20260924 05:10:26.433887 11624 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:10:26.433909 11624 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4404b88dd534a11bae5a17c9a912074, State: Initialized, Role: FOLLOWER
I20260924 05:10:26.434051 11624 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4404b88dd534a11bae5a17c9a912074" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37865 } }
I20260924 05:10:26.434128 11624 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:10:26.434186 11624 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:10:26.434233 11624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:10:26.434752 11624 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4404b88dd534a11bae5a17c9a912074" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37865 } }
I20260924 05:10:26.434903 11624 leader_election.cc:302] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4404b88dd534a11bae5a17c9a912074; no voters:
I20260924 05:10:26.435050 11624 leader_election.cc:288] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:10:26.435096 11629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:10:26.435204 11629 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [term 1 LEADER]: Becoming Leader. State: Replica: a4404b88dd534a11bae5a17c9a912074, State: Running, Role: LEADER
I20260924 05:10:26.435320 11624 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:10:26.435364 11629 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4404b88dd534a11bae5a17c9a912074" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37865 } }
I20260924 05:10:26.435698 11630 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4404b88dd534a11bae5a17c9a912074. Latest consensus state: current_term: 1 leader_uuid: "a4404b88dd534a11bae5a17c9a912074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4404b88dd534a11bae5a17c9a912074" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37865 } } }
I20260924 05:10:26.435796 11630 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [sys.catalog]: This master's current role is: LEADER
I20260924 05:10:26.436009 11635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:10:26.436708 11635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:10:26.436923 11631 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4404b88dd534a11bae5a17c9a912074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4404b88dd534a11bae5a17c9a912074" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37865 } } }
I20260924 05:10:26.437002 11624 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 05:10:26.437397 11631 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074 [sys.catalog]: This master's current role is: LEADER
Sep 24 05:10:26 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:10:26.441669 11635 catalog_manager.cc:1347] Generated new cluster ID: a6e2557185584b16bed14e009b6ade6a
I20260924 05:10:26.441737 11635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:10:26.445034 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:26.431974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.0.98.62:37865)
0924 05:10:26.432170 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:26.432178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:26.432201 (+ 23us) server_negotiation.cc:407] Connection header received
0924 05:10:26.432606 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:26.432612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:26.432658 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:26.432806 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:10:26.433429 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:26.434570 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:26.435602 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:26.436304 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:26.436420 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:26.439999 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:10:26.440046 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:10:26.440049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:10:26.440125 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:10:26.442987 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:26.443427 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:26.443432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:26.443432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:26.443542 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:26.443750 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:26.443751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:26.443751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:26.444075 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:10:26.444225 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:26.444318 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 05:10:26.444587 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260924 05:10:26.450544 11635 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:10:26.451347 11635 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:10:26.455165 11635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074: Generated new TSK 0
I20260924 05:10:26.455327 11635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:10:27.129546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37865
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:10:27.238072 11669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:10:27.238332 11669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:10:27.238365 11669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:10:27.240231 11669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:10:27.240299 11669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:10:27.240339 11669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:10:27.242115 11669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:50011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:10:27.242368 11669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:10:27.242626 11669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:10:27.247798 11674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:10:27.247807 11677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:10:27.247974 11675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:10:27.248379 11669 server_base.cc:1034] running on GCE node
I20260924 05:10:27.248603 11669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:10:27.248878 11669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:10:27.250074 11669 hybrid_clock.cc:648] HybridClock initialized: now 1790226627250059 us; error 28 us; skew 500 ppm
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:10:27.253095 11669 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:10:27.253612 11669 webserver.cc:460] Webserver started at http://127.0.98.1:38189/ using document root <none> and password file <none>
I20260924 05:10:27.253903 11669 fs_manager.cc:359] Metadata directory not provided
I20260924 05:10:27.253985 11669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:10:27.254123 11669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:10:27.267442 11669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "4490e8c5748c4fa080a90a1999bb9216"
format_stamp: "Formatted at 2026-09-24 05:10:27 on dist-test-slave-mr2f"
server_key: "51ef26725b71e178af86e92e6da8fc57"
server_key_iv: "6dceb55913cba84cb9dbbf897c961f37"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:27.268103 11669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4490e8c5748c4fa080a90a1999bb9216"
format_stamp: "Formatted at 2026-09-24 05:10:27 on dist-test-slave-mr2f"
server_key: "51ef26725b71e178af86e92e6da8fc57"
server_key_iv: "6dceb55913cba84cb9dbbf897c961f37"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:27.270519 11669 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 05:10:27.286938 11686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:27.287179 11669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 05:10:27.287299 11669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "4490e8c5748c4fa080a90a1999bb9216"
format_stamp: "Formatted at 2026-09-24 05:10:27 on dist-test-slave-mr2f"
server_key: "51ef26725b71e178af86e92e6da8fc57"
server_key_iv: "6dceb55913cba84cb9dbbf897c961f37"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:27.287437 11669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:10:27.305716 11669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:10:27.308465 11669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:10:27.308681 11669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:10:27.308975 11669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:10:27.309391 11669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:10:27.309485 11669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:27.309571 11669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:10:27.309619 11669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:27.317585 11669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33333
I20260924 05:10:27.317689 11799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33333 every 8 connection(s)
I20260924 05:10:27.318066 11669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:10:27.325599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 11669
I20260924 05:10:27.325709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:10:27.325904 392 external_mini_cluster.cc:1428] Setting key 7bc50c58715bcb5285acc3044782d67d
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:10:27.333097 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.318810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37865 (local address 127.0.98.1:59273)
0924 05:10:27.319075 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:27.319117 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:10:27.319715 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:10:27.323631 (+ 3916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:10:27.323637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:10:27.323902 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:10:27.324217 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.324224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.325303 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.325304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.326592 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.326595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.327163 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.327165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.327300 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.327900 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:10:27.327910 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:10:27.329483 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:10:27.331542 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:10:27.331548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:10:27.331551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:10:27.331821 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:10:27.332033 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:10:27.332034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:10:27.332034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:10:27.332159 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:10:27.332441 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:10:27.332448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:10:27.332626 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 05:10:27.332730 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 05:10:27.332873 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":78,"threads_started":1}
I20260924 05:10:27.333411 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.318857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59273 (local address 127.0.98.62:37865)
0924 05:10:27.323145 (+ 4288us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:27.323147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:27.323158 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:10:27.323186 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:27.323187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:27.323200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:27.323282 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:10:27.324346 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.325191 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.326686 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.327037 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.327098 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.329599 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:10:27.329600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:10:27.329601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:10:27.329608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:10:27.331445 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:27.331901 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:27.331902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:27.331903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:27.331963 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:27.332229 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:27.332230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:27.332231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:27.332359 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:10:27.332409 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:27.333236 (+ 827us) server_negotiation.cc:299] Negotiation successful
0924 05:10:27.333316 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4255,"thread_start_us":80,"threads_started":1}
I20260924 05:10:27.333972 11800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37865
I20260924 05:10:27.334084 11800 heartbeater.cc:464] Registering TS with master...
I20260924 05:10:27.334312 11800 heartbeater.cc:511] Master 127.0.98.62:37865 requested a full tablet report, sending...
I20260924 05:10:27.334856 11588 ts_manager.cc:194] Registered new tserver with Master: 4490e8c5748c4fa080a90a1999bb9216 (127.0.98.1:33333)
I20260924 05:10:27.335807 11588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59273
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:10:27.378720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37865
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 05:10:27.506640 11808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:10:27.506985 11808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:10:27.507053 11808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:10:27.508769 11808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:10:27.508862 11808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:10:27.508940 11808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:10:27.510982 11808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:50011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:10:27.511472 11808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:10:27.511751 11808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:10:27.518090 11814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:10:27.518388 11813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:10:27.518477 11816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:10:27.519609 11808 server_base.cc:1034] running on GCE node
I20260924 05:10:27.519814 11808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:10:27.520056 11808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:10:27.521227 11808 hybrid_clock.cc:648] HybridClock initialized: now 1790226627521200 us; error 41 us; skew 500 ppm
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:10:27.524389 11808 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:10:27.524868 11808 webserver.cc:460] Webserver started at http://127.0.98.2:44797/ using document root <none> and password file <none>
I20260924 05:10:27.525060 11808 fs_manager.cc:359] Metadata directory not provided
I20260924 05:10:27.525098 11808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:10:27.525182 11808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:10:27.537770 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "bab54f59b748414d985f938b3a8923d5"
format_stamp: "Formatted at 2026-09-24 05:10:27 on dist-test-slave-mr2f"
server_key: "9155aa77632aac88f29430f9a11559d8"
server_key_iv: "90262217e545f133106702d7f8e88621"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:27.538192 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bab54f59b748414d985f938b3a8923d5"
format_stamp: "Formatted at 2026-09-24 05:10:27 on dist-test-slave-mr2f"
server_key: "9155aa77632aac88f29430f9a11559d8"
server_key_iv: "90262217e545f133106702d7f8e88621"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:27.548913 11808 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.000s
I20260924 05:10:27.562793 11824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:27.563028 11808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 05:10:27.563112 11808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "bab54f59b748414d985f938b3a8923d5"
format_stamp: "Formatted at 2026-09-24 05:10:27 on dist-test-slave-mr2f"
server_key: "9155aa77632aac88f29430f9a11559d8"
server_key_iv: "90262217e545f133106702d7f8e88621"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:27.563216 11808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:10:27.579825 11808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:10:27.582317 11808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:10:27.582520 11808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:10:27.582803 11808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:10:27.583233 11808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:10:27.583355 11808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:27.583434 11808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:10:27.583492 11808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:27.590309 11808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36435
I20260924 05:10:27.593400 11937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36435 every 8 connection(s)
I20260924 05:10:27.593701 11808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:10:27.595168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 11808
I20260924 05:10:27.595284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:10:27.595489 392 external_mini_cluster.cc:1428] Setting key bb7f805d490086a2d8be1ad38b3f73f2
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:10:27.606426 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.595091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34533 (local address 127.0.98.62:37865)
0924 05:10:27.595142 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:27.595144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:27.595424 (+ 280us) server_negotiation.cc:407] Connection header received
0924 05:10:27.596189 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:27.596195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:27.596207 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:27.596256 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:10:27.597075 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.597853 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.599416 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.599727 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.599769 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.602949 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:10:27.602950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:10:27.602951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:10:27.602959 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:10:27.604711 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:27.605209 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:27.605211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:27.605211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:27.605266 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:27.605584 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:27.605585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:27.605586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:27.605728 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:10:27.605788 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:27.606240 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 05:10:27.606330 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:10:27.607241 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.594806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37865 (local address 127.0.98.2:34533)
0924 05:10:27.595189 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:27.595291 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0924 05:10:27.596044 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:10:27.596438 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:10:27.596442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:10:27.596704 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:10:27.596961 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.596968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.597977 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.597978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.599310 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.599313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.599859 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.599860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.600244 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.600858 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:10:27.600868 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:10:27.602822 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:10:27.604808 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:10:27.604815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:10:27.604817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:10:27.605114 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:10:27.605356 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:10:27.605357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:10:27.605358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:10:27.605487 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:10:27.605881 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:10:27.605887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:10:27.606063 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 05:10:27.606806 (+ 743us) client_negotiation.cc:240] Negotiation successful
0924 05:10:27.606989 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":80,"threads_started":1}
I20260924 05:10:27.607925 11938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37865
I20260924 05:10:27.608016 11938 heartbeater.cc:464] Registering TS with master...
I20260924 05:10:27.608232 11938 heartbeater.cc:511] Master 127.0.98.62:37865 requested a full tablet report, sending...
I20260924 05:10:27.608640 11588 ts_manager.cc:194] Registered new tserver with Master: bab54f59b748414d985f938b3a8923d5 (127.0.98.2:36435)
I20260924 05:10:27.609164 11588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34533
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:10:27.653301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37865
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:10:27.765136 11943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:10:27.765338 11943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:10:27.765362 11943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:10:27.766510 11943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:10:27.766556 11943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:10:27.766582 11943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:10:27.768169 11943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:50011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:10:27.768390 11943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:10:27.768601 11943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:10:27.773804 11949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:10:27.773944 11948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:10:27.773804 11951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:10:27.774503 11943 server_base.cc:1034] running on GCE node
I20260924 05:10:27.774708 11943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:10:27.774995 11943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:10:27.776304 11943 hybrid_clock.cc:648] HybridClock initialized: now 1790226627776208 us; error 113 us; skew 500 ppm
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:10:27.779098 11943 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:10:27.781565 11943 webserver.cc:460] Webserver started at http://127.0.98.3:38389/ using document root <none> and password file <none>
I20260924 05:10:27.781834 11943 fs_manager.cc:359] Metadata directory not provided
I20260924 05:10:27.781908 11943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:10:27.782043 11943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:10:27.796408 11943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "75ae88a03ced4473837d14aa6f0eac57"
format_stamp: "Formatted at 2026-09-24 05:10:27 on dist-test-slave-mr2f"
server_key: "2766d958c79032c906d0ed8f4f322a6a"
server_key_iv: "1102f825b42edc32b15b30a34685fd1f"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:27.796976 11943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "75ae88a03ced4473837d14aa6f0eac57"
format_stamp: "Formatted at 2026-09-24 05:10:27 on dist-test-slave-mr2f"
server_key: "2766d958c79032c906d0ed8f4f322a6a"
server_key_iv: "1102f825b42edc32b15b30a34685fd1f"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:27.799347 11943 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 05:10:27.817437 11958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:27.817746 11943 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 05:10:27.817863 11943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "75ae88a03ced4473837d14aa6f0eac57"
format_stamp: "Formatted at 2026-09-24 05:10:27 on dist-test-slave-mr2f"
server_key: "2766d958c79032c906d0ed8f4f322a6a"
server_key_iv: "1102f825b42edc32b15b30a34685fd1f"
server_key_version: "kuduclusterkey@0"
I20260924 05:10:27.817962 11943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:10:27.836561 11943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:10:27.839058 11943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:10:27.839233 11943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:10:27.839531 11943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:10:27.839943 11943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:10:27.840024 11943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:27.840088 11943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:10:27.840137 11943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:10:27.847649 11943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43761
I20260924 05:10:27.847767 12071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43761 every 8 connection(s)
I20260924 05:10:27.848086 11943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:10:27.852526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 11943
I20260924 05:10:27.852644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:10:27.852815 392 external_mini_cluster.cc:1428] Setting key 0d4cf372edba18e32cfac7a565180040
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:10:27.858645 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.848950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33917 (local address 127.0.98.62:37865)
0924 05:10:27.848994 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:27.848997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:27.849640 (+ 643us) server_negotiation.cc:407] Connection header received
0924 05:10:27.850256 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:27.850258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:27.850270 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:27.850317 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:10:27.850892 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.851381 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.852991 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.853276 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.853361 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.855815 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:10:27.855816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:10:27.855816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:10:27.855824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:10:27.857256 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:27.857666 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:27.857667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:27.857668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:27.857727 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:27.857976 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:27.857977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:27.857977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:27.858097 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:10:27.858171 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:27.858465 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 05:10:27.858549 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:10:27.858872 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.848951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37865 (local address 127.0.98.3:33917)
0924 05:10:27.849105 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:27.849145 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:10:27.850140 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:10:27.850396 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:10:27.850399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:10:27.850578 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:10:27.850812 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.850818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.851512 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.851513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.852303 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.852305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.853378 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.853379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.853759 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.854330 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:10:27.854339 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:10:27.855689 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:10:27.857354 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:10:27.857361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:10:27.857363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:10:27.857578 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:10:27.857793 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:10:27.857794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:10:27.857795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:10:27.857909 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:10:27.858163 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:10:27.858169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:10:27.858329 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 05:10:27.858445 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 05:10:27.858585 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260924 05:10:27.859342 12072 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37865
I20260924 05:10:27.859457 12072 heartbeater.cc:464] Registering TS with master...
I20260924 05:10:27.859634 12072 heartbeater.cc:511] Master 127.0.98.62:37865 requested a full tablet report, sending...
I20260924 05:10:27.859990 11588 ts_manager.cc:194] Registered new tserver with Master: 75ae88a03ced4473837d14aa6f0eac57 (127.0.98.3:43761)
I20260924 05:10:27.860370 11588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33917
I20260924 05:10:27.864499 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:10:27.874867 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.866129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.0.98.62:37865)
0924 05:10:27.866171 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:27.866173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:27.866442 (+ 269us) server_negotiation.cc:407] Connection header received
0924 05:10:27.867214 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:27.867215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:27.867225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:27.867304 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:10:27.867931 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.868475 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.869980 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.870187 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.870271 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.871854 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:10:27.871855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:10:27.871856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:10:27.871864 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:10:27.873603 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:27.874059 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:27.874060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:27.874060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:27.874121 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:10:27.874314 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:10:27.874315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:10:27.874316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:10:27.874433 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:10:27.874543 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:27.874644 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 05:10:27.874732 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:10:27.878118 11588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:10:27.880064 11588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:10:27.889736 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.882638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43761 (local address 127.0.0.1:58568)
0924 05:10:27.885449 (+ 2811us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:27.885458 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:10:27.885502 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:10:27.885772 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:10:27.885773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:10:27.885780 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:10:27.885821 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.885822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.887100 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.887101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.888420 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.888422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.889435 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.889436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.889509 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.889522 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:10:27.889588 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 05:10:27.889641 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2777,"thread_start_us":64,"threads_started":1}
I20260924 05:10:27.890012 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.883190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.0.98.3:43761)
0924 05:10:27.883569 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:27.883574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:27.885562 (+ 1988us) server_negotiation.cc:407] Connection header received
0924 05:10:27.885608 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:27.885611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:27.885626 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:27.885695 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:10:27.885898 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.887004 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.888527 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.889272 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.889336 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.889402 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:27.889860 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 05:10:27.889904 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":90,"threads_started":1}
I20260924 05:10:27.890988 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.882407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36435 (local address 127.0.0.1:48050)
0924 05:10:27.883633 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:27.883677 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 05:10:27.883744 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:10:27.884124 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:10:27.884127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:10:27.884129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:10:27.884193 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.884195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.887772 (+ 3577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.887773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.888818 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.888821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.890734 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.890735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.890826 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.890830 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:10:27.890874 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:10:27.890913 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"thread_start_us":79,"threads_started":1}
I20260924 05:10:27.890926 12006 tablet_service.cc:1467] Processing CreateTablet for tablet ccbc49e8232c4baf91c93a139bb60a5d (DEFAULT_TABLE table=test-table [id=c73c8baefd7c40b09b00d2b831904c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:10:27.891289 12006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccbc49e8232c4baf91c93a139bb60a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:10:27.891479 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.882952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.0.98.2:36435)
0924 05:10:27.883897 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:27.883901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:27.883911 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:10:27.883949 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:27.883951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:27.883967 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:27.884036 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:10:27.884292 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.887374 (+ 3082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.888932 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.889633 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.889687 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.889750 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:27.891358 (+ 1608us) server_negotiation.cc:299] Negotiation successful
0924 05:10:27.891402 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":71,"threads_started":1}
I20260924 05:10:27.892051 11872 tablet_service.cc:1467] Processing CreateTablet for tablet ccbc49e8232c4baf91c93a139bb60a5d (DEFAULT_TABLE table=test-table [id=c73c8baefd7c40b09b00d2b831904c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:10:27.892379 11872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccbc49e8232c4baf91c93a139bb60a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:10:27.894958 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.882817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33333 (local address 127.0.0.1:39656)
0924 05:10:27.890627 (+ 7810us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:27.890637 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:10:27.890677 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:10:27.891303 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:10:27.891303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:10:27.891306 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:10:27.891342 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.891343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.892615 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.892616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.893782 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:27.893785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.894739 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.894740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:27.894811 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.894816 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:10:27.894861 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:10:27.894895 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6968,"thread_start_us":70,"threads_started":1}
I20260924 05:10:27.895841 12091 tablet_bootstrap.cc:492] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: Bootstrap starting.
I20260924 05:10:27.895985 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:27.882924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.0.98.1:33333)
0924 05:10:27.891014 (+ 8090us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:27.891019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:27.891032 (+ 13us) server_negotiation.cc:407] Connection header received
0924 05:10:27.891087 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:27.891090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:27.891105 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:27.891237 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:10:27.891412 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.892526 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.893876 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:27.894556 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:27.894625 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:27.894705 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:27.895862 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0924 05:10:27.895919 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8053,"thread_start_us":57,"threads_started":1}
I20260924 05:10:27.896109 12090 tablet_bootstrap.cc:492] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: Bootstrap starting.
I20260924 05:10:27.896690 11716 tablet_service.cc:1467] Processing CreateTablet for tablet ccbc49e8232c4baf91c93a139bb60a5d (DEFAULT_TABLE table=test-table [id=c73c8baefd7c40b09b00d2b831904c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:10:27.897003 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccbc49e8232c4baf91c93a139bb60a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:10:27.897089 12091 tablet_bootstrap.cc:654] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: Neither blocks nor log segments found. Creating new log.
I20260924 05:10:27.897375 12091 log.cc:824] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:10:27.898083 12091 tablet_bootstrap.cc:492] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: No bootstrap required, opened a new log
I20260924 05:10:27.898140 12091 ts_tablet_manager.cc:1397] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:10:27.900205 12091 raft_consensus.cc:348] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } }
I20260924 05:10:27.900390 12091 raft_consensus.cc:374] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:10:27.900430 12091 raft_consensus.cc:729] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab54f59b748414d985f938b3a8923d5, State: Initialized, Role: FOLLOWER
I20260924 05:10:27.900538 12091 consensus_queue.cc:260] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } }
I20260924 05:10:27.900801 12091 ts_tablet_manager.cc:1428] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:10:27.901028 11938 heartbeater.cc:503] Master 127.0.98.62:37865 was elected leader, sending a full tablet report...
I20260924 05:10:27.902309 12090 tablet_bootstrap.cc:654] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: Neither blocks nor log segments found. Creating new log.
I20260924 05:10:27.902613 12090 log.cc:824] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: Log is configured to *not* fsync() on all Append() calls
I20260924 05:10:27.903002 12093 tablet_bootstrap.cc:492] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216: Bootstrap starting.
I20260924 05:10:27.903574 12093 tablet_bootstrap.cc:654] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216: Neither blocks nor log segments found. Creating new log.
I20260924 05:10:27.903821 12093 log.cc:824] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216: Log is configured to *not* fsync() on all Append() calls
I20260924 05:10:27.904517 12093 tablet_bootstrap.cc:492] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216: No bootstrap required, opened a new log
I20260924 05:10:27.904603 12093 ts_tablet_manager.cc:1397] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:10:27.906718 12093 raft_consensus.cc:348] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } }
I20260924 05:10:27.906944 12093 raft_consensus.cc:374] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:10:27.907047 12093 raft_consensus.cc:729] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4490e8c5748c4fa080a90a1999bb9216, State: Initialized, Role: FOLLOWER
I20260924 05:10:27.907254 12093 consensus_queue.cc:260] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } }
I20260924 05:10:27.907357 12090 tablet_bootstrap.cc:492] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: No bootstrap required, opened a new log
I20260924 05:10:27.907451 12090 ts_tablet_manager.cc:1397] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260924 05:10:27.907600 12093 ts_tablet_manager.cc:1428] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:10:27.907740 11800 heartbeater.cc:503] Master 127.0.98.62:37865 was elected leader, sending a full tablet report...
I20260924 05:10:27.909442 12090 raft_consensus.cc:348] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } }
I20260924 05:10:27.909657 12090 raft_consensus.cc:374] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:10:27.909756 12090 raft_consensus.cc:729] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ae88a03ced4473837d14aa6f0eac57, State: Initialized, Role: FOLLOWER
I20260924 05:10:27.909987 12090 consensus_queue.cc:260] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } }
I20260924 05:10:27.910291 12090 ts_tablet_manager.cc:1428] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 05:10:27.910444 12072 heartbeater.cc:503] Master 127.0.98.62:37865 was elected leader, sending a full tablet report...
Sep 24 05:10:27 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 05:10:28.073964 11801 tablet.cc:2369] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:10:28.094678 11939 tablet.cc:2369] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:10:28.098945 12073 tablet.cc:2369] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:10:28.158051 12098 raft_consensus.cc:482] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:10:28.158215 12098 raft_consensus.cc:504] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } }
I20260924 05:10:28.158641 12098 leader_election.cc:288] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bab54f59b748414d985f938b3a8923d5 (127.0.98.2:36435), 4490e8c5748c4fa080a90a1999bb9216 (127.0.98.1:33333)
I20260924 05:10:28.163491 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:28.158914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33333 (local address 127.0.98.3:39173)
0924 05:10:28.159013 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:28.159025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:10:28.159082 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:10:28.159754 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:10:28.159755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:10:28.159757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:10:28.159824 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:28.159826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.160787 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.160788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:28.161936 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:28.161940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.163230 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.163231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:28.163305 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:28.163309 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:10:28.163365 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:10:28.163421 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 05:10:28.163772 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:28.159555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39173 (local address 127.0.98.1:33333)
0924 05:10:28.159597 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:28.159599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:28.159607 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:10:28.159636 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:28.159637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:28.159648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:28.159702 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:10:28.159895 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.160664 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.162064 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.163117 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.163187 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:28.163198 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:28.163664 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 05:10:28.163705 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:10:28.164032 11754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccbc49e8232c4baf91c93a139bb60a5d" candidate_uuid: "75ae88a03ced4473837d14aa6f0eac57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4490e8c5748c4fa080a90a1999bb9216" is_pre_election: true
I20260924 05:10:28.164209 11754 raft_consensus.cc:2454] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75ae88a03ced4473837d14aa6f0eac57 in term 0.
I20260924 05:10:28.164549 11961 leader_election.cc:302] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4490e8c5748c4fa080a90a1999bb9216, 75ae88a03ced4473837d14aa6f0eac57; no voters:
I20260924 05:10:28.164741 12098 raft_consensus.cc:2781] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:10:28.164795 12098 raft_consensus.cc:482] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:10:28.164819 12098 raft_consensus.cc:3037] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:10:28.165812 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:28.159456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46625 (local address 127.0.98.2:36435)
0924 05:10:28.159496 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:28.159498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:28.161717 (+ 2219us) server_negotiation.cc:407] Connection header received
0924 05:10:28.161825 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:28.161827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:28.161838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:28.161867 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:10:28.162082 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.162813 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.164816 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.165363 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.165423 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:28.165434 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:28.165656 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 05:10:28.165725 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:10:28.165800 12098 raft_consensus.cc:504] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } }
I20260924 05:10:28.165925 12098 leader_election.cc:288] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [CANDIDATE]: Term 1 election: Requested vote from peers bab54f59b748414d985f938b3a8923d5 (127.0.98.2:36435), 4490e8c5748c4fa080a90a1999bb9216 (127.0.98.1:33333)
I20260924 05:10:28.166045 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:28.159253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36435 (local address 127.0.98.3:46625)
0924 05:10:28.159756 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:28.159769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:10:28.161750 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:10:28.161930 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:10:28.161930 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:10:28.161932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:10:28.162006 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:28.162008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.163040 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.163041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:28.164715 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:10:28.164717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.165510 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.165511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:10:28.165575 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:28.165578 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:10:28.165914 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 05:10:28.165954 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":110,"threads_started":1}
I20260924 05:10:28.166513 11876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccbc49e8232c4baf91c93a139bb60a5d" candidate_uuid: "75ae88a03ced4473837d14aa6f0eac57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab54f59b748414d985f938b3a8923d5" is_pre_election: true
I20260924 05:10:28.166666 11876 raft_consensus.cc:2454] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75ae88a03ced4473837d14aa6f0eac57 in term 0.
I20260924 05:10:28.166806 11889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccbc49e8232c4baf91c93a139bb60a5d" candidate_uuid: "75ae88a03ced4473837d14aa6f0eac57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab54f59b748414d985f938b3a8923d5"
I20260924 05:10:28.166882 11889 raft_consensus.cc:3037] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:10:28.167729 11889 raft_consensus.cc:2454] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75ae88a03ced4473837d14aa6f0eac57 in term 1.
I20260924 05:10:28.168156 11960 leader_election.cc:302] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75ae88a03ced4473837d14aa6f0eac57, bab54f59b748414d985f938b3a8923d5; no voters:
I20260924 05:10:28.168352 12098 raft_consensus.cc:2781] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:10:28.168383 11754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccbc49e8232c4baf91c93a139bb60a5d" candidate_uuid: "75ae88a03ced4473837d14aa6f0eac57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4490e8c5748c4fa080a90a1999bb9216"
I20260924 05:10:28.168507 11754 raft_consensus.cc:3037] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:10:28.168543 12098 raft_consensus.cc:686] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 1 LEADER]: Becoming Leader. State: Replica: 75ae88a03ced4473837d14aa6f0eac57, State: Running, Role: LEADER
I20260924 05:10:28.168634 12098 consensus_queue.cc:237] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } }
I20260924 05:10:28.169339 11754 raft_consensus.cc:2454] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75ae88a03ced4473837d14aa6f0eac57 in term 1.
I20260924 05:10:28.169759 11588 catalog_manager.cc:5495] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75ae88a03ced4473837d14aa6f0eac57 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "75ae88a03ced4473837d14aa6f0eac57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75ae88a03ced4473837d14aa6f0eac57" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 } health_report { overall_health: UNKNOWN } } }
Sep 24 05:10:28 dist-test-slave-mr2f krb5kdc[11112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:10:28.244974 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:28.241624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.0.98.3:43761)
0924 05:10:28.241751 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 05:10:28.241754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:10:28.241763 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:10:28.241850 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:10:28.241851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:10:28.241906 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:10:28.242037 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:10:28.242333 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.243095 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.244050 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:10:28.244323 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:10:28.244386 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:10:28.244606 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:10:28.244742 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:10:28.244837 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 05:10:28.244860 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 05:10:28.244882 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 05:10:28.246867 11876 raft_consensus.cc:1261] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [term 1 FOLLOWER]: Refusing update from remote peer 75ae88a03ced4473837d14aa6f0eac57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:10:28.246896 11754 raft_consensus.cc:1261] T ccbc49e8232c4baf91c93a139bb60a5d P 4490e8c5748c4fa080a90a1999bb9216 [term 1 FOLLOWER]: Refusing update from remote peer 75ae88a03ced4473837d14aa6f0eac57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:10:28.247213 12098 consensus_queue.cc:1035] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab54f59b748414d985f938b3a8923d5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:10:28.247442 12100 consensus_queue.cc:1035] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4490e8c5748c4fa080a90a1999bb9216" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:10:28.255389 12104 mvcc.cc:205] Tried to move back new op lower bound from 7332768269295714304 to 7332768268978905088. Current Snapshot: MvccSnapshot[applied={T|T < 7332768269295714304}]
I20260924 05:10:28.255630 12107 mvcc.cc:205] Tried to move back new op lower bound from 7332768269295714304 to 7332768268978905088. Current Snapshot: MvccSnapshot[applied={T|T < 7332768269295714304}]
I20260924 05:10:28.263353 11588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:10:28.263453 11588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:10:28.264377 11588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4404b88dd534a11bae5a17c9a912074: Sending DeleteTablet for 3 replicas of tablet ccbc49e8232c4baf91c93a139bb60a5d
I20260924 05:10:28.264765 11872 tablet_service.cc:1514] Processing DeleteTablet for tablet ccbc49e8232c4baf91c93a139bb60a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:10:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48050
I20260924 05:10:28.264788 11716 tablet_service.cc:1514] Processing DeleteTablet for tablet ccbc49e8232c4baf91c93a139bb60a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:10:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39656
I20260924 05:10:28.265201 12006 tablet_service.cc:1514] Processing DeleteTablet for tablet ccbc49e8232c4baf91c93a139bb60a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:10:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58568
I20260924 05:10:28.265792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 11669
W20260924 05:10:28.274435 11574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33333 (error 108)
I20260924 05:10:28.274498 12115 tablet_replica.cc:330] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: stopping tablet replica
I20260924 05:10:28.274595 12115 raft_consensus.cc:2229] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:10:28.274731 12115 raft_consensus.cc:2258] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:10:28.274888 12113 tablet_replica.cc:330] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: stopping tablet replica
I20260924 05:10:28.275048 12113 raft_consensus.cc:2229] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:10:28.275204 12113 raft_consensus.cc:2258] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:10:28.275272 12115 ts_tablet_manager.cc:1905] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:10:28.275722 12113 ts_tablet_manager.cc:1905] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:10:28.277077 12115 ts_tablet_manager.cc:1918] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:10:28.277213 12115 log.cc:1196] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/ccbc49e8232c4baf91c93a139bb60a5d
I20260924 05:10:28.277390 12113 ts_tablet_manager.cc:1918] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:10:28.277550 12115 ts_tablet_manager.cc:1939] T ccbc49e8232c4baf91c93a139bb60a5d P 75ae88a03ced4473837d14aa6f0eac57: Deleting consensus metadata
I20260924 05:10:28.277977 11574 catalog_manager.cc:4841] TS 75ae88a03ced4473837d14aa6f0eac57 (127.0.98.3:43761): tablet ccbc49e8232c4baf91c93a139bb60a5d (table test-table [id=c73c8baefd7c40b09b00d2b831904c59]) successfully deleted
I20260924 05:10:28.278127 12113 log.cc:1196] T ccbc49e8232c4baf91c93a139bb60a5d P bab54f59b748414d985f938b3a8923d5: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/ccbc49e8232c4baf91c93a139bb60a5d
I20260924 05:10:28.278306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 11808
I20260924 05:10:28.279362 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:28.279258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33333 (local address 127.0.0.1:39662)
0924 05:10:28.279301 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:28.279330 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:10:28.279517 11574 catalog_manager.cc:4584] TS 4490e8c5748c4fa080a90a1999bb9216 (127.0.98.1:33333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccbc49e8232c4baf91c93a139bb60a5d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33333: connect: Connection refused (error 111)
I20260924 05:10:28.284435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 11943
I20260924 05:10:28.286288 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:10:28.286168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36435 (local address 127.0.0.1:48056)
0924 05:10:28.286231 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 05:10:28.286260 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 05:10:28.289902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 11556
I20260924 05:10:28.297802 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 05:10:28.642050 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 05:10:28.642153 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 05:10:28.987185 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 05:10:28.987 UTC [11142] LOG: received smart shutdown request
2026-09-24 05:10:28.990 UTC [11142] LOG: background worker "logical replication launcher" (PID 11151) exited with exit code 1
2026-09-24 05:10:28.990 UTC [11146] LOG: shutting down
2026-09-24 05:10:29.141 UTC [11142] LOG: database system is shut down
2026-09-24T05:10:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44460 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:10:29 dist-test-slave-mr2f krb5kdc[12126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:10:29 dist-test-slave-mr2f krb5kdc[12126](info): set up 2 sockets
Sep 24 05:10:29 dist-test-slave-mr2f krb5kdc[12126](info): commencing operation
krb5kdc: starting...
W20260924 05:10:30.270505 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:10:30 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:10:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:10:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 05:10:30.420991 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhdUPbm/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/postgres -l logfile start
2026-09-24 05:10:32.763 UTC [12156] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 05:10:32.763 UTC [12156] LOG: listening on IPv4 address "127.0.98.20", port 37925
2026-09-24 05:10:32.766 UTC [12156] LOG: listening on Unix socket "/tmp/.s.PGSQL.37925"
2026-09-24 05:10:32.799 UTC [12159] LOG: database system was shut down at 2026-09-24 05:10:31 UTC
2026-09-24 05:10:32.803 UTC [12156] LOG: database system is ready to accept connections
I20260924 05:10:33.738469 392 mini_postgres.cc:96] Postgres bound to 37925
2026-09-24 05:10:33.744 UTC [12167] FATAL: database "slave" does not exist
127.0.98.20:37925 - accepting connections
I20260924 05:10:33.744656 392 mini_ranger.cc:161] Starting Ranger...
I20260924 05:10:33.758045 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 05:10:34.188599 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 05:10:34.188683 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-admin
2026-09-24 05:10:34,452 [I] DB FLAVOR :POSTGRES
2026-09-24 05:10:34,452 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:10:34,452 [I] Checking connection..
2026-09-24 05:10:34,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 05:10:34,690 [I] Checking connection passed.
2026-09-24 05:10:34,690 [I] --------- Verifying version history table ---------
2026-09-24 05:10:34,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:10:34,922 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:10:34,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:10:35,167 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:10:35,167 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 05:10:35,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 05:10:35.409 UTC [12231] WARNING: there is no transaction in progress
2026-09-24 05:10:35,416 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 05:10:35,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:10:35,655 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 05:10:35,655 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:10:35,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 05:10:35,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 05:10:36,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mr2f.c.gcp-upstream.internal','N') ;"
2026-09-24 05:10:36,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:10:36,602 [I] Table x_portal_user does not exist in database ranger
2026-09-24 05:10:36,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:10:36,854 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 05:10:36,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:10:37,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:10:37,356 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 05:10:37,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 05:10:38.278 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.293 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.307 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.321 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.337 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.356 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.419 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.425 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.439 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.448 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.458 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.471 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.480 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.486 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.493 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:38.499 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 05:10:39,073 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 05:10:39,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 05:10:39,318 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 05:10:39,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:10:39,538 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 05:10:39,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:10:39,761 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 05:10:39,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:10:39,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:10:40,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mr2f.c.gcp-upstream.internal';"
2026-09-24 05:10:40,438 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 05:10:40,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 05:10:40,685 [I] DB_PATCHES have already been applied
I20260924 05:10:40.690892 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhdUPbm/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 05:10:40.690949 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 05:10:40.693853 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52173
Sep 24, 2026 5:10:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:10:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 5:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 5:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 5:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 5:10:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 05:10:41.801239 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.057s user 0.002s sys 0.006s
Sep 24, 2026 5:11:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:11:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 5:11:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:11:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:11:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 05:11:02.736656 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 05:11:02 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226662, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 05:11:02.818049 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 05:11:02.826540 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 05:11:03.616508 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 05:11:03.616645 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-kms
I20260924 05:11:03.616688 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:37925
2026-09-24 05:11:03,790 [I] DB FLAVOR :POSTGRES
2026-09-24 05:11:03,790 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:11:03,790 [I] Checking connection
2026-09-24 05:11:03,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 05:11:04,023 [I] connection success
2026-09-24 05:11:04,023 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 05:11:04,023 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 05:11:04,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 05:11:04,275 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 05:11:04,276 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:11:04,276 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 05:11:04,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 05:11:04,512 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 05:11:04.931123 392 mini_ranger_kms.cc:310] Created kms service
I20260924 05:11:05.114483 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 05:11:05.156461 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 05:11:05 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226662, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:05.518286 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 05:11:05.518417 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhdUPbm/build/release/bin/postgresql.jar:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhdUPbm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 05:11:05.518435 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 05:11:05.521728 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12519
I20260924 05:11:05.521836 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:44355
Sep 24, 2026 5:11:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:11:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:11:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:11:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:11:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:11:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 05:11:07 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:07.743203 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.925s user 0.000s sys 0.147s
I20260924 05:11:07.743299 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 05:11:07.743345 392 mini_ranger_kms.cc:401] 127.0.98.20:44355/kms/v1/keys
Sep 24 05:11:07 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:11:08 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:11:10 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:11:10.303817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40955
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40955
--ranger_config_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44355/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:10.404911 12570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:10.405107 12570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:10.405128 12570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:10.406167 12570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:10.406207 12570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:10.406224 12570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:10.406235 12570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:10.406248 12570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:10.407707 12570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:44355/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40955
--ranger_config_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:10.407929 12570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:10.408136 12570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:10.412958 12575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:10.413015 12576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:10.413014 12578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:10.413028 12570 server_base.cc:1034] running on GCE node
I20260924 05:11:10.413337 12570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:10.413582 12570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:10.414709 12570 hybrid_clock.cc:648] HybridClock initialized: now 1790226670414691 us; error 29 us; skew 500 ppm
Sep 24 05:11:10 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:10.428262 12570 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:11:10.428632 12570 webserver.cc:460] Webserver started at http://127.0.98.62:35951/ using document root <none> and password file <none>
I20260924 05:11:10.428800 12570 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:10.428834 12570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:10.428917 12570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:11:10 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:10.487010 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "b5b92226a2d5497fb9f4a7a650b64042"
format_stamp: "Formatted at 2026-09-24 05:11:10 on dist-test-slave-mr2f"
server_key: "d11ccdd1278067d9fd6413dc96ea8d13"
server_key_iv: "8626d769df461057fe1ea38efdbecce0"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:10.487406 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5b92226a2d5497fb9f4a7a650b64042"
format_stamp: "Formatted at 2026-09-24 05:11:10 on dist-test-slave-mr2f"
server_key: "d11ccdd1278067d9fd6413dc96ea8d13"
server_key_iv: "8626d769df461057fe1ea38efdbecce0"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:10.488874 12570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:11:10.503441 12585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:10.503655 12570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:11:10.503749 12570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "b5b92226a2d5497fb9f4a7a650b64042"
format_stamp: "Formatted at 2026-09-24 05:11:10 on dist-test-slave-mr2f"
server_key: "d11ccdd1278067d9fd6413dc96ea8d13"
server_key_iv: "8626d769df461057fe1ea38efdbecce0"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:10.503857 12570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:10.518699 12570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:10.520627 12570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:10.520773 12570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:10.525437 12570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40955
I20260924 05:11:10.525462 12637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40955 every 8 connection(s)
I20260924 05:11:10.525832 12570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:11:10.526515 12638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:10.528855 12638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Bootstrap starting.
I20260924 05:11:10.529441 12638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:10.529443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 12570
I20260924 05:11:10.529544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:11:10.529733 392 external_mini_cluster.cc:1428] Setting key fb36e7fb0daa4df3d74e39f6bcc0a739
I20260924 05:11:10.529780 12638 log.cc:824] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:10.530514 12638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: No bootstrap required, opened a new log
I20260924 05:11:10.533174 12638 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } }
I20260924 05:11:10.533370 12638 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:10.533433 12638 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b92226a2d5497fb9f4a7a650b64042, State: Initialized, Role: FOLLOWER
I20260924 05:11:10.533579 12638 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } }
I20260924 05:11:10.533664 12638 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:11:10.533691 12638 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:11:10.533753 12638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:10.534444 12638 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } }
I20260924 05:11:10.534582 12638 leader_election.cc:302] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5b92226a2d5497fb9f4a7a650b64042; no voters:
I20260924 05:11:10.534742 12638 leader_election.cc:288] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:11:10.534965 12644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:10.535041 12638 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:10.535153 12644 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 1 LEADER]: Becoming Leader. State: Replica: b5b92226a2d5497fb9f4a7a650b64042, State: Running, Role: LEADER
I20260924 05:11:10.535310 12644 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } }
I20260924 05:11:10.535749 12644 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5b92226a2d5497fb9f4a7a650b64042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } }
I20260924 05:11:10.535893 12644 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:10.536069 12644 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5b92226a2d5497fb9f4a7a650b64042. Latest consensus state: current_term: 1 leader_uuid: "b5b92226a2d5497fb9f4a7a650b64042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } }
I20260924 05:11:10.536183 12644 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: This master's current role is: LEADER
Sep 24 05:11:10 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:10.536458 12652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:11:10.536527 12638 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 05:11:10.536957 12652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:11:10.544577 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:10.530171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.0.98.62:40955)
0924 05:11:10.530326 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:10.530335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:10.530366 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:11:10.530921 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:10.530928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:10.530974 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:10.531115 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:10.531706 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:10.532617 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:10.533365 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:10.533995 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:10.534130 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:10.537036 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:10.537040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:10.537040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:10.537050 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:10.542110 (+ 5060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:10.542783 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:10.542786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:10.542786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:10.542914 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:10.543123 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:10.543124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:10.543124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:10.543473 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:10.543592 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:10.544070 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 05:11:10.544234 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260924 05:11:10.548497 12652 catalog_manager.cc:1347] Generated new cluster ID: c99e755e7166498b80138ddda0de28da
I20260924 05:11:10.548557 12652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:11:10.576117 12652 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:11:10.578100 12652 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:11:10.585947 12652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Generated new TSK 0
I20260924 05:11:10.586231 12652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:11:11.235453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40955
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:11.335603 12683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:11.335894 12683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:11.335958 12683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:11.337134 12683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:11.337174 12683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:11.337200 12683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:11:11.338696 12683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:44355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:11.338999 12683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:11.339227 12683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:11.344240 12689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:11.344445 12691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:11.344630 12683 server_base.cc:1034] running on GCE node
W20260924 05:11:11.344722 12688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:11.344998 12683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:11.345281 12683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:11.346462 12683 hybrid_clock.cc:648] HybridClock initialized: now 1790226671346442 us; error 33 us; skew 500 ppm
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:11.349500 12683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:11:11.349989 12683 webserver.cc:460] Webserver started at http://127.0.98.1:43937/ using document root <none> and password file <none>
I20260924 05:11:11.350275 12683 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:11.350342 12683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:11.350508 12683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:11.363776 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "2a5f004edddd456ea83de97d5e73621e"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "c7783a878c338102c11a736c6aabd21f"
server_key_iv: "d11457ea1971fc9e6df9055cefe28fdf"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:11.364214 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2a5f004edddd456ea83de97d5e73621e"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "c7783a878c338102c11a736c6aabd21f"
server_key_iv: "d11457ea1971fc9e6df9055cefe28fdf"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:11.365924 12683 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 05:11:11.386942 12698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:11.387151 12683 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260924 05:11:11.387260 12683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "2a5f004edddd456ea83de97d5e73621e"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "c7783a878c338102c11a736c6aabd21f"
server_key_iv: "d11457ea1971fc9e6df9055cefe28fdf"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:11.387373 12683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:11.400557 12683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:11.402540 12683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:11.402711 12683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:11.402992 12683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:11.403362 12683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:11.403411 12683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:11.403443 12683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:11.403503 12683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:11.410959 12683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34793
I20260924 05:11:11.411314 12683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:11:11.412120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 12683
I20260924 05:11:11.412230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:11:11.412408 392 external_mini_cluster.cc:1428] Setting key ed5210ada619ab28eb3059464081f835
I20260924 05:11:11.414863 12813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34793 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 05:11:11.425844 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:11.415380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40955 (local address 127.0.98.1:53083)
0924 05:11:11.415630 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:11.415677 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:11.416357 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:11.416682 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:11.416686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:11.416954 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:11.417242 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:11.417248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.418256 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.418257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:11.419514 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:11.419517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.420027 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.420028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:11.420154 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:11.420740 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:11.420750 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:11.422322 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:11.424166 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:11.424173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:11.424175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:11.424464 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:11.424694 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:11.424695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:11.424695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:11.424840 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:11.425146 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:11.425152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:11.425318 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 05:11:11.425432 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 05:11:11.425587 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260924 05:11:11.426170 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:11.415409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53083 (local address 127.0.98.62:40955)
0924 05:11:11.415557 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:11.415559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:11.415801 (+ 242us) server_negotiation.cc:407] Connection header received
0924 05:11:11.416488 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:11.416489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:11.416503 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:11.416571 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:11.417361 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.418149 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.419615 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.419886 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.419928 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:11.422430 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:11.422431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:11.422432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:11.422438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:11.424075 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:11.424555 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:11.424556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:11.424557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:11.424624 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:11.424920 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:11.424921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:11.424921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:11.425054 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:11.425119 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:11.425982 (+ 863us) server_negotiation.cc:299] Negotiation successful
0924 05:11:11.426073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260924 05:11:11.426579 12814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40955
I20260924 05:11:11.426739 12814 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:11.427007 12814 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
I20260924 05:11:11.427573 12602 ts_manager.cc:194] Registered new tserver with Master: 2a5f004edddd456ea83de97d5e73621e (127.0.98.1:34793)
I20260924 05:11:11.428468 12602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:11:11.442250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40955
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:11.554260 12820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:11.554584 12820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:11.554638 12820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:11.556345 12820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:11.556399 12820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:11.556439 12820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:11.558571 12820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:44355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:11.558825 12820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:11.559069 12820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:11.565140 12828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:11.565145 12830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:11.565487 12827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:11.566267 12820 server_base.cc:1034] running on GCE node
I20260924 05:11:11.566484 12820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:11.566740 12820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:11.567936 12820 hybrid_clock.cc:648] HybridClock initialized: now 1790226671567915 us; error 32 us; skew 500 ppm
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:11.570996 12820 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:11:11.571535 12820 webserver.cc:460] Webserver started at http://127.0.98.2:42543/ using document root <none> and password file <none>
I20260924 05:11:11.571833 12820 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:11.571943 12820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:11.572108 12820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:11.586258 12820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "c88cbd4878aa4840a35ca9a3b9ea6667"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "164cb633b42fd84e518b3176164d8588"
server_key_iv: "29dbf4ad4ed226af7d1517018f1eca21"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:11.586836 12820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c88cbd4878aa4840a35ca9a3b9ea6667"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "164cb633b42fd84e518b3176164d8588"
server_key_iv: "29dbf4ad4ed226af7d1517018f1eca21"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:11.589982 12820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 05:11:11.606592 12837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:11.606957 12820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 05:11:11.607156 12820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "c88cbd4878aa4840a35ca9a3b9ea6667"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "164cb633b42fd84e518b3176164d8588"
server_key_iv: "29dbf4ad4ed226af7d1517018f1eca21"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:11.607355 12820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:11.618930 12820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:11.620886 12820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:11.621044 12820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:11.621271 12820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:11.621654 12820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:11.621707 12820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:11.621770 12820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:11.621798 12820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:11.629246 12820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39021
I20260924 05:11:11.629769 12820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:11:11.630685 12950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39021 every 8 connection(s)
I20260924 05:11:11.634984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 12820
I20260924 05:11:11.635067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:11:11.635252 392 external_mini_cluster.cc:1428] Setting key 3c669c199e05f2647ba11b5c3c67afa2
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:11.642727 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:11.630710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47481 (local address 127.0.98.62:40955)
0924 05:11:11.630862 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:11.630865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:11.631472 (+ 607us) server_negotiation.cc:407] Connection header received
0924 05:11:11.632094 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:11.632095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:11.632107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:11.632170 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:11.633077 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.633884 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.635782 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.636052 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.636101 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:11.638924 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:11.638925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:11.638925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:11.638932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:11.640735 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:11.641232 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:11.641233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:11.641234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:11.641357 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:11.641633 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:11.641634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:11.641634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:11.641830 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:11.641915 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:11.642469 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 05:11:11.642588 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:11:11.643739 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:11.630964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40955 (local address 127.0.98.2:47481)
0924 05:11:11.631176 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:11.631232 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:11.631972 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:11.632294 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:11.632299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:11.632598 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:11.632940 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:11.632946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.634010 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.634011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:11.635303 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:11.635306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.636346 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.636347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:11.636489 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:11.637087 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:11.637097 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:11.638820 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:11.640833 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:11.640839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:11.640841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:11.641163 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:11.641409 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:11.641410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:11.641410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:11.641538 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:11.641964 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:11.641970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:11.642231 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 05:11:11.643257 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0924 05:11:11.643473 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":106,"threads_started":1}
I20260924 05:11:11.645852 12951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40955
I20260924 05:11:11.645995 12951 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:11.646260 12951 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
I20260924 05:11:11.646657 12602 ts_manager.cc:194] Registered new tserver with Master: c88cbd4878aa4840a35ca9a3b9ea6667 (127.0.98.2:39021)
I20260924 05:11:11.647188 12602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47481
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:11:11.670157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40955
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:11.787123 12957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:11.787400 12957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:11.787436 12957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:11.789146 12957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:11.789206 12957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:11.789247 12957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:11:11.791375 12957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:44355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:11.791617 12957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:11.791921 12957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:11.798161 12965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:11.798179 12962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:11.798514 12963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:11.798540 12957 server_base.cc:1034] running on GCE node
I20260924 05:11:11.798815 12957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:11.799101 12957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:11.806495 12957 hybrid_clock.cc:648] HybridClock initialized: now 1790226671806461 us; error 38 us; skew 500 ppm
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:11.809747 12957 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:11:11.810261 12957 webserver.cc:460] Webserver started at http://127.0.98.3:46039/ using document root <none> and password file <none>
I20260924 05:11:11.810576 12957 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:11.810663 12957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:11.810833 12957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:11.829440 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "20eed2d42cd5409389df19c3f31cc733"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "5f391d9846bf079027246620ee9e4b9f"
server_key_iv: "7aa4f77a9fecc43e26bef30c8f1bc8e2"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:11.830003 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "20eed2d42cd5409389df19c3f31cc733"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "5f391d9846bf079027246620ee9e4b9f"
server_key_iv: "7aa4f77a9fecc43e26bef30c8f1bc8e2"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:11.833495 12957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 05:11:11.859022 12972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:11.861333 12957 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260924 05:11:11.861459 12957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "20eed2d42cd5409389df19c3f31cc733"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "5f391d9846bf079027246620ee9e4b9f"
server_key_iv: "7aa4f77a9fecc43e26bef30c8f1bc8e2"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:11.861568 12957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:11.870147 12957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:11.872341 12957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:11.872505 12957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:11.872777 12957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:11.873176 12957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:11.873230 12957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:11.873268 12957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:11.873294 12957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:11.880316 12957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34585
I20260924 05:11:11.880705 12957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:11:11.881731 13085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34585 every 8 connection(s)
Sep 24 05:11:11 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:11.888334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 12957
I20260924 05:11:11.888444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:11:11.888664 392 external_mini_cluster.cc:1428] Setting key 751337b26c952dba0d0e4c0ac4b461b5
I20260924 05:11:11.893020 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:11.882405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57901 (local address 127.0.98.62:40955)
0924 05:11:11.882453 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:11.882456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:11.882668 (+ 212us) server_negotiation.cc:407] Connection header received
0924 05:11:11.883397 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:11.883399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:11.883412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:11.883444 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:11.884270 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.884961 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.886127 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.886342 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.886418 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:11.889786 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:11.889788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:11.889788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:11.889796 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:11.891470 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:11.891932 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:11.891933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:11.891933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:11.891994 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:11.892314 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:11.892315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:11.892316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:11.892435 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:11.892512 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:11.892843 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 05:11:11.892921 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:11:11.893222 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:11.882390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40955 (local address 127.0.98.3:57901)
0924 05:11:11.882543 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:11.882585 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:11.883261 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:11.883630 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:11.883634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:11.883895 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:11.884158 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:11.884164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.885087 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.885087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:11.886036 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:11.886038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:11.886449 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:11.886450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:11.886561 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:11.887160 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:11.887170 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:11.889682 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:11.891558 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:11.891564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:11.891567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:11.891799 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:11.892060 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:11.892060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:11.892061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:11.892222 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:11.892502 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:11.892508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:11.892695 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 05:11:11.892822 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 05:11:11.892978 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 05:11:11.893913 13086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40955
I20260924 05:11:11.894021 13086 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:11.894199 13086 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
I20260924 05:11:11.894577 12602 ts_manager.cc:194] Registered new tserver with Master: 20eed2d42cd5409389df19c3f31cc733 (127.0.98.3:34585)
I20260924 05:11:11.895007 12602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57901
I20260924 05:11:11.900225 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:11:11.900931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 12683
I20260924 05:11:11.906260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 12820
I20260924 05:11:11.912729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 12957
I20260924 05:11:11.918534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 12570
I20260924 05:11:11.923591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40955
--webserver_interface=127.0.98.62
--webserver_port=35951
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40955
--ranger_config_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44355/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44355/kms with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:12.001497 13089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:12.001710 13089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:12.001732 13089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:12.002750 13089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:12.002832 13089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:12.002857 13089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:12.002883 13089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:12.002913 13089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:12.004424 13089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44355/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40955
--ranger_config_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:12.004703 13089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:12.004900 13089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:12.009462 13097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:12.009481 13095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:12.009586 13094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:12.009744 13089 server_base.cc:1034] running on GCE node
I20260924 05:11:12.010052 13089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:12.010339 13089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:12.011475 13089 hybrid_clock.cc:648] HybridClock initialized: now 1790226672011460 us; error 30 us; skew 500 ppm
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:12.013886 13089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:11:12.014240 13089 webserver.cc:460] Webserver started at http://127.0.98.62:35951/ using document root <none> and password file <none>
I20260924 05:11:12.014408 13089 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:12.014454 13089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:12.079907 13089 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:12.080587 13105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.080791 13089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:12.080884 13089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "b5b92226a2d5497fb9f4a7a650b64042"
format_stamp: "Formatted at 2026-09-24 05:11:10 on dist-test-slave-mr2f"
server_key: "d11ccdd1278067d9fd6413dc96ea8d13"
server_key_iv: "8626d769df461057fe1ea38efdbecce0"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:12.081240 13089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:12.099731 13089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:12.101725 13089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:12.101883 13089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:12.105873 13089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40955
I20260924 05:11:12.105917 13157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40955 every 8 connection(s)
I20260924 05:11:12.106253 13089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:11:12.107411 13158 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:11:12.108055 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Bootstrap starting.
I20260924 05:11:12.108793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 13089
I20260924 05:11:12.109073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34793
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40955
--webserver_port=43937
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44355/kms with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:12.116585 13158 log.cc:824] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:12.118182 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:11:12.118467 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Bootstrap complete.
I20260924 05:11:12.120707 13158 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } }
I20260924 05:11:12.120951 13158 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b92226a2d5497fb9f4a7a650b64042, State: Initialized, Role: FOLLOWER
I20260924 05:11:12.121143 13158 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } }
I20260924 05:11:12.121251 13158 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:11:12.121304 13158 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:11:12.121351 13158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:11:12.122345 13158 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } }
I20260924 05:11:12.122469 13158 leader_election.cc:302] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5b92226a2d5497fb9f4a7a650b64042; no voters:
I20260924 05:11:12.122628 13158 leader_election.cc:288] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:11:12.122661 13162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:11:12.122776 13162 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [term 2 LEADER]: Becoming Leader. State: Replica: b5b92226a2d5497fb9f4a7a650b64042, State: Running, Role: LEADER
I20260924 05:11:12.122857 13162 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } }
I20260924 05:11:12.122891 13158 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:12.123119 13163 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b5b92226a2d5497fb9f4a7a650b64042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } }
I20260924 05:11:12.123227 13163 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:12.123368 13164 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5b92226a2d5497fb9f4a7a650b64042. Latest consensus state: current_term: 2 leader_uuid: "b5b92226a2d5497fb9f4a7a650b64042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b92226a2d5497fb9f4a7a650b64042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40955 } } }
I20260924 05:11:12.123442 13164 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042 [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:12.123507 13167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:11:12.124023 13167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:11:12.124207 13167 catalog_manager.cc:1259] Loaded cluster ID: c99e755e7166498b80138ddda0de28da
I20260924 05:11:12.124286 13167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:11:12.124378 13158 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 05:11:12.128047 13167 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:11:12.128533 13167 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Loaded TSK: 0
I20260924 05:11:12.128769 13167 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:11:12.192826 13160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:12.193087 13160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:12.193125 13160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:12.194697 13160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:12.194751 13160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:12.194814 13160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:11:12.196954 13160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43937
--tserver_master_addrs=127.0.98.62:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:12.197191 13160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:12.197413 13160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:12.203166 13198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:12.203315 13195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:12.203631 13196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:12.203634 13160 server_base.cc:1034] running on GCE node
I20260924 05:11:12.203917 13160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:12.204213 13160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:12.205361 13160 hybrid_clock.cc:648] HybridClock initialized: now 1790226672205361 us; error 36 us; skew 500 ppm
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:12.208334 13160 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:11:12.208807 13160 webserver.cc:460] Webserver started at http://127.0.98.1:43937/ using document root <none> and password file <none>
I20260924 05:11:12.209021 13160 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:12.209097 13160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:12.238780 13160 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260924 05:11:12.239770 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.239995 13160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:12.240110 13160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "2a5f004edddd456ea83de97d5e73621e"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "c7783a878c338102c11a736c6aabd21f"
server_key_iv: "d11457ea1971fc9e6df9055cefe28fdf"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:12.240651 13160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:12.267975 13160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:12.269915 13160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:12.270057 13160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:12.270289 13160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:12.270649 13160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:12.270704 13160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.270740 13160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:12.270764 13160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.277634 13160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34793
I20260924 05:11:12.277756 13319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34793 every 8 connection(s)
I20260924 05:11:12.278052 13160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:12.287420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 13160
I20260924 05:11:12.287731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39021
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40955
--webserver_port=42543
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44355/kms with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:12.293812 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:12.278749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40955 (local address 127.0.98.1:51821)
0924 05:11:12.279024 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:12.279062 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:12.279650 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:12.280404 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:12.280407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:12.280626 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:12.280870 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:12.280876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.282507 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.282507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:12.283396 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:12.283398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.284244 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.284245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:12.284364 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:12.284906 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:12.284916 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:12.286508 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:12.291864 (+ 5356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:12.291869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:12.291871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:12.292167 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:12.292439 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:12.292440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:12.292440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:12.292573 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:12.293089 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:12.293095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:12.293256 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 05:11:12.293367 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 05:11:12.293522 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":74,"threads_started":1}
I20260924 05:11:12.294803 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:12.278836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51821 (local address 127.0.98.62:40955)
0924 05:11:12.279974 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:12.279983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:12.280019 (+ 36us) server_negotiation.cc:407] Connection header received
0924 05:11:12.280153 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:12.280159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:12.280215 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:12.280990 (+ 775us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:12.281108 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.282389 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.283508 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.284016 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.284125 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:12.286648 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:12.286653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:12.286653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:12.286664 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:12.291770 (+ 5106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:12.292258 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:12.292261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:12.292262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:12.292364 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:12.292650 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:12.292651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:12.292651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:12.292978 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:12.293060 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:12.294244 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0924 05:11:12.294369 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":88,"threads_started":1}
I20260924 05:11:12.295893 13320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40955
I20260924 05:11:12.296152 13320 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:12.296389 13320 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
W20260924 05:11:12.297289 13320 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40955 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:11:12.297361 13320 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:12.297430 13320 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
I20260924 05:11:12.297855 13320 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:12.297932 13320 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
W20260924 05:11:12.298269 13320 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:11:12.396961 13324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:12.397224 13324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:12.397257 13324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:12.398875 13324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:12.398926 13324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:12.398964 13324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:11:12.401072 13324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42543
--tserver_master_addrs=127.0.98.62:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:12.401300 13324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:12.401536 13324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:12.406613 13330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:12.406700 13329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:12.406617 13332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:12.407337 13324 server_base.cc:1034] running on GCE node
I20260924 05:11:12.407527 13324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:12.407783 13324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:12.408946 13324 hybrid_clock.cc:648] HybridClock initialized: now 1790226672408925 us; error 39 us; skew 500 ppm
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:12.412570 13324 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:11:12.413069 13324 webserver.cc:460] Webserver started at http://127.0.98.2:42543/ using document root <none> and password file <none>
I20260924 05:11:12.413303 13324 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:12.413400 13324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:12.430394 13324 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260924 05:11:12.431185 13340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.431380 13324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:11:12.431473 13324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "c88cbd4878aa4840a35ca9a3b9ea6667"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "164cb633b42fd84e518b3176164d8588"
server_key_iv: "29dbf4ad4ed226af7d1517018f1eca21"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:12.431890 13324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:12.460122 13324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:12.462131 13324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:12.462283 13324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:12.462514 13324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:12.462915 13324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:12.462968 13324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.463001 13324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:12.463020 13324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.470216 13324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39021
I20260924 05:11:12.470693 13324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:11:12.473747 13453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39021 every 8 connection(s)
I20260924 05:11:12.474972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 13324
I20260924 05:11:12.475229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34585
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40955
--webserver_port=46039
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44355/kms with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:12.489674 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:12.471501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33157 (local address 127.0.98.62:40955)
0924 05:11:12.471547 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:12.471548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:12.471838 (+ 290us) server_negotiation.cc:407] Connection header received
0924 05:11:12.472501 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:12.472503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:12.472515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:12.472545 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:12.473328 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.474069 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.483406 (+ 9337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.483702 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.483756 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:12.486286 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:12.486288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:12.486288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:12.486297 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:12.488029 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:12.488535 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:12.488537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:12.488537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:12.488599 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:12.488908 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:12.488909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:12.488910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:12.489042 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:12.489090 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:12.489489 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 05:11:12.489576 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:11:12.490388 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:12.471495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40955 (local address 127.0.98.2:33157)
0924 05:11:12.471694 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:12.471755 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:12.472370 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:12.472726 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:12.472730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:12.472950 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:12.473212 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:12.473218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.474180 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.474181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:12.483270 (+ 9089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:12.483275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.483887 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.483888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:12.484028 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:12.484668 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:12.484678 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:12.486166 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:12.488127 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:12.488134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:12.488136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:12.488437 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:12.488672 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:12.488672 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:12.488673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:12.488805 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:12.489143 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:12.489149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:12.489319 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 05:11:12.489994 (+ 675us) client_negotiation.cc:240] Negotiation successful
0924 05:11:12.490167 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260924 05:11:12.491084 13454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40955
I20260924 05:11:12.491176 13454 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:12.491356 13454 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
W20260924 05:11:12.492044 13454 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40955 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:11:12.492111 13454 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:12.492166 13454 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
I20260924 05:11:12.492514 13454 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:12.492586 13454 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
W20260924 05:11:12.492931 13454 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:11:12.576408 13457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:12.576695 13457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:12.576731 13457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:12.578389 13457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:12.578444 13457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:12.578481 13457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:11:12.580426 13457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46039
--tserver_master_addrs=127.0.98.62:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:12.580653 13457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:12.580927 13457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:12.587064 13465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:12.587086 13463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:12.587064 13462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:12.587563 13457 server_base.cc:1034] running on GCE node
I20260924 05:11:12.587764 13457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:12.588002 13457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:12.589217 13457 hybrid_clock.cc:648] HybridClock initialized: now 1790226672589178 us; error 41 us; skew 500 ppm
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:12.592046 13457 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:11:12.592537 13457 webserver.cc:460] Webserver started at http://127.0.98.3:46039/ using document root <none> and password file <none>
I20260924 05:11:12.592799 13457 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:12.592916 13457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:12.607066 13457 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 05:11:12.607782 13473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.607985 13457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:12.608067 13457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "20eed2d42cd5409389df19c3f31cc733"
format_stamp: "Formatted at 2026-09-24 05:11:11 on dist-test-slave-mr2f"
server_key: "5f391d9846bf079027246620ee9e4b9f"
server_key_iv: "7aa4f77a9fecc43e26bef30c8f1bc8e2"
server_key_version: "kuduclusterkey@0"
I20260924 05:11:12.608413 13457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:12.616873 13457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:12.618942 13457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:12.619091 13457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:12.619313 13457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:12.619655 13457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:12.619704 13457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.619740 13457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:12.619763 13457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:12.626520 13457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34585
I20260924 05:11:12.626585 13587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34585 every 8 connection(s)
I20260924 05:11:12.626993 13457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:11:12.632498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 13457
Sep 24 05:11:12 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:12.638260 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:12.627742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33199 (local address 127.0.98.62:40955)
0924 05:11:12.627787 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:12.627790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:12.628111 (+ 321us) server_negotiation.cc:407] Connection header received
0924 05:11:12.628743 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:12.628745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:12.628758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:12.628831 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:12.629605 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.630378 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.631903 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.632201 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.632249 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:12.634746 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:12.634748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:12.634748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:12.634755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:12.636507 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:12.637088 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:12.637090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:12.637090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:12.637153 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:12.637458 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:12.637459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:12.637459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:12.637590 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:12.637637 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:12.638072 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 05:11:12.638161 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:11:12.638700 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:12.627737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40955 (local address 127.0.98.3:33199)
0924 05:11:12.627971 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:12.628008 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:12.628616 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:12.628984 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:12.628989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:12.629214 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:12.629490 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:12.629496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.630489 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.630490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:12.631790 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:12.631793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.632362 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.632362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:12.632497 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:12.633066 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:12.633076 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:12.634635 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:12.636647 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:12.636653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:12.636655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:12.636982 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:12.637244 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:12.637245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:12.637245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:12.637375 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:12.637720 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:12.637727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:12.637901 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 05:11:12.638017 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 05:11:12.638196 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260924 05:11:12.639150 13588 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40955
I20260924 05:11:12.639251 13588 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:12.639430 13588 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
W20260924 05:11:12.640056 13588 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40955 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:11:12.640136 13588 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:12.640199 13588 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
I20260924 05:11:12.640479 13588 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:12.640550 13588 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
W20260924 05:11:12.640829 13588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:11:12.644955 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:12.632903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.0.98.62:40955)
0924 05:11:12.638045 (+ 5142us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:12.638047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:12.638053 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:11:12.638073 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:12.638073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:12.638083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:12.638141 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:12.638946 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.639695 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.640756 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:12.641022 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:12.641075 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:12.642393 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:12.642394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:12.642394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:12.642400 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:12.643787 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:12.644178 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:12.644179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:12.644180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:12.644256 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:12.644452 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:12.644453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:12.644454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:12.644558 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:12.644742 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:12.644777 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 05:11:12.644853 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5124,"thread_start_us":59,"threads_started":1}
Sep 24 05:11:13 dist-test-slave-mr2f krb5kdc[12126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:13.298444 13320 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:13.298630 13320 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
I20260924 05:11:13.298979 13122 ts_manager.cc:194] Registered new tserver with Master: 2a5f004edddd456ea83de97d5e73621e (127.0.98.1:34793)
I20260924 05:11:13.299707 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51821
I20260924 05:11:13.493134 13454 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:13.493274 13454 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
I20260924 05:11:13.493516 13122 ts_manager.cc:194] Registered new tserver with Master: c88cbd4878aa4840a35ca9a3b9ea6667 (127.0.98.2:39021)
I20260924 05:11:13.494027 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33157
Sep 24 05:11:13 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:13.641014 13588 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:13.641148 13588 heartbeater.cc:511] Master 127.0.98.62:40955 requested a full tablet report, sending...
I20260924 05:11:13.641547 13122 ts_manager.cc:194] Registered new tserver with Master: 20eed2d42cd5409389df19c3f31cc733 (127.0.98.3:34585)
I20260924 05:11:13.642031 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33199
I20260924 05:11:13.650403 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:11:13.670122 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.659036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.0.98.62:40955)
0924 05:11:13.659331 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:13.659332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:13.659338 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:11:13.659354 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:13.659355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:13.659363 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:13.659439 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:13.659966 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.660438 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.661174 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.661501 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.662246 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.667500 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:13.667502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:13.667502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:13.667511 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:13.669012 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:13.669369 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:13.669371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:13.669371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:13.669430 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:13.669619 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:13.669620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:13.669621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:13.669729 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:13.669833 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:13.669936 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 05:11:13.670019 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":67,"threads_started":1}
I20260924 05:11:13.672120 13122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:11:13.673779 13122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:11:13.682001 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.677148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34793 (local address 127.0.0.1:49420)
0924 05:11:13.677528 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:13.677564 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:13.677627 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:13.677965 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:13.677968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:13.677970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:13.678026 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.678027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.679248 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.679249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.680476 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.680479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.681736 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.681737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.681812 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.681823 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 05:11:13.681886 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:11:13.681928 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":75,"threads_started":1}
I20260924 05:11:13.682596 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.677390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.0.98.1:34793)
0924 05:11:13.677756 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:13.677760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:13.677769 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:11:13.677814 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:13.677816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:13.677830 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:13.677893 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:13.678093 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.679159 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.680570 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.681564 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.681627 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.681704 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:13.682473 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 05:11:13.682523 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":50,"threads_started":1}
I20260924 05:11:13.683343 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 25e905371a8840fe97384c7e51d529e2 (DEFAULT_TABLE table=test-table [id=fd658a71f94d4ab2a2f7ba11a02505ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:13.683696 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e905371a8840fe97384c7e51d529e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:13.685822 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.677160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39021 (local address 127.0.0.1:48368)
0924 05:11:13.680962 (+ 3802us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:13.680972 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:13.681029 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:13.681289 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:13.681290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:13.681292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:13.681329 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.681331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.682898 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.682899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.684306 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.684309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.685602 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.685603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.685660 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.685664 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:13.685726 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 05:11:13.685758 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3788,"thread_start_us":88,"threads_started":1}
I20260924 05:11:13.686122 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.677392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.98.2:39021)
0924 05:11:13.680913 (+ 3521us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:13.680917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:13.681061 (+ 144us) server_negotiation.cc:407] Connection header received
0924 05:11:13.681103 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:13.681105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:13.681121 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:13.681207 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:13.681404 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.682815 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.684399 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.685445 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.685508 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.685569 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:13.686001 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 05:11:13.686043 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3486,"thread_start_us":54,"threads_started":1}
I20260924 05:11:13.686715 13388 tablet_service.cc:1467] Processing CreateTablet for tablet 25e905371a8840fe97384c7e51d529e2 (DEFAULT_TABLE table=test-table [id=fd658a71f94d4ab2a2f7ba11a02505ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:13.687063 13388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e905371a8840fe97384c7e51d529e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:13.687446 13622 tablet_bootstrap.cc:492] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e: Bootstrap starting.
I20260924 05:11:13.688076 13622 tablet_bootstrap.cc:654] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:13.688419 13622 log.cc:824] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:13.692507 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.677579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34585 (local address 127.0.0.1:46164)
0924 05:11:13.682367 (+ 4788us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:13.682376 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:13.682405 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:13.688926 (+ 6521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:13.688927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:13.688928 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:13.688957 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.688959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.690273 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.690274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.691409 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.691411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.692324 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.692325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.692382 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.692385 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:11:13.692412 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 05:11:13.692447 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4779,"thread_start_us":60,"threads_started":1}
I20260924 05:11:13.692776 13624 tablet_bootstrap.cc:492] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: Bootstrap starting.
I20260924 05:11:13.693010 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.677763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.0.98.3:34585)
0924 05:11:13.688762 (+ 10999us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:13.688766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:13.688777 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:11:13.688810 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:13.688812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:13.688827 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:13.689016 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:13.689086 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.690181 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.691506 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.692234 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.692788 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.692874 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:13.692899 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 05:11:13.692937 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10963,"thread_start_us":69,"threads_started":1}
I20260924 05:11:13.693434 13624 tablet_bootstrap.cc:654] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:13.693681 13516 tablet_service.cc:1467] Processing CreateTablet for tablet 25e905371a8840fe97384c7e51d529e2 (DEFAULT_TABLE table=test-table [id=fd658a71f94d4ab2a2f7ba11a02505ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:13.693764 13624 log.cc:824] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:13.693957 13516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e905371a8840fe97384c7e51d529e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:13.694562 13622 tablet_bootstrap.cc:492] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e: No bootstrap required, opened a new log
I20260924 05:11:13.694994 13622 ts_tablet_manager.cc:1397] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260924 05:11:13.697103 13626 tablet_bootstrap.cc:492] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: Bootstrap starting.
I20260924 05:11:13.697732 13626 tablet_bootstrap.cc:654] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:13.697760 13622 raft_consensus.cc:348] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } }
I20260924 05:11:13.697985 13622 raft_consensus.cc:374] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:13.698058 13622 raft_consensus.cc:729] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5f004edddd456ea83de97d5e73621e, State: Initialized, Role: FOLLOWER
I20260924 05:11:13.698199 13626 log.cc:824] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:13.698227 13622 consensus_queue.cc:260] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } }
I20260924 05:11:13.698621 13622 ts_tablet_manager.cc:1428] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 05:11:13.698827 13320 heartbeater.cc:503] Master 127.0.98.62:40955 was elected leader, sending a full tablet report...
I20260924 05:11:13.698988 13626 tablet_bootstrap.cc:492] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: No bootstrap required, opened a new log
I20260924 05:11:13.699051 13626 ts_tablet_manager.cc:1397] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:11:13.701469 13626 raft_consensus.cc:348] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } }
I20260924 05:11:13.701628 13626 raft_consensus.cc:374] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:13.701666 13626 raft_consensus.cc:729] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20eed2d42cd5409389df19c3f31cc733, State: Initialized, Role: FOLLOWER
I20260924 05:11:13.701797 13626 consensus_queue.cc:260] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } }
I20260924 05:11:13.702013 13626 ts_tablet_manager.cc:1428] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:11:13.702047 13588 heartbeater.cc:503] Master 127.0.98.62:40955 was elected leader, sending a full tablet report...
I20260924 05:11:13.706805 13624 tablet_bootstrap.cc:492] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: No bootstrap required, opened a new log
I20260924 05:11:13.706875 13624 ts_tablet_manager.cc:1397] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20260924 05:11:13.709236 13624 raft_consensus.cc:348] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } }
I20260924 05:11:13.709399 13624 raft_consensus.cc:374] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:13.709427 13624 raft_consensus.cc:729] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88cbd4878aa4840a35ca9a3b9ea6667, State: Initialized, Role: FOLLOWER
I20260924 05:11:13.709547 13624 consensus_queue.cc:260] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } }
I20260924 05:11:13.709770 13624 ts_tablet_manager.cc:1428] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:11:13.709807 13454 heartbeater.cc:503] Master 127.0.98.62:40955 was elected leader, sending a full tablet report...
W20260924 05:11:13.724634 13455 tablet.cc:2369] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:11:13.780827 13321 tablet.cc:2369] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 05:11:13 dist-test-slave-mr2f krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:11:13.838891 13629 raft_consensus.cc:482] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:13.839039 13629 raft_consensus.cc:504] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } }
I20260924 05:11:13.839438 13629 leader_election.cc:288] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a5f004edddd456ea83de97d5e73621e (127.0.98.1:34793), c88cbd4878aa4840a35ca9a3b9ea6667 (127.0.98.2:39021)
I20260924 05:11:13.844623 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.839760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35887 (local address 127.0.98.1:34793)
0924 05:11:13.839816 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:13.839818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:13.839894 (+ 76us) server_negotiation.cc:407] Connection header received
0924 05:11:13.840002 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:13.840003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:13.840015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:13.840045 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:13.840466 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.841093 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.843415 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.844032 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.844085 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.844095 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:13.844504 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 05:11:13.844546 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:11:13.845010 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.839551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34793 (local address 127.0.98.3:35887)
0924 05:11:13.839832 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:13.839842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:13.839924 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:13.840287 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:13.840288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:13.840291 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:13.840385 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.840387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.841541 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.841542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.843316 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.843318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.844345 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.844346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.844411 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.844414 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:11:13.844878 (+ 464us) client_negotiation.cc:240] Negotiation successful
0924 05:11:13.844921 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260924 05:11:13.845409 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e905371a8840fe97384c7e51d529e2" candidate_uuid: "20eed2d42cd5409389df19c3f31cc733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5f004edddd456ea83de97d5e73621e" is_pre_election: true
I20260924 05:11:13.845607 13274 raft_consensus.cc:2454] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20eed2d42cd5409389df19c3f31cc733 in term 0.
I20260924 05:11:13.845847 13477 leader_election.cc:302] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20eed2d42cd5409389df19c3f31cc733, 2a5f004edddd456ea83de97d5e73621e; no voters:
I20260924 05:11:13.845983 13629 raft_consensus.cc:2781] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:11:13.846040 13629 raft_consensus.cc:482] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:11:13.846063 13629 raft_consensus.cc:3037] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:13.846737 13629 raft_consensus.cc:504] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } }
I20260924 05:11:13.846967 13629 leader_election.cc:288] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [CANDIDATE]: Term 1 election: Requested vote from peers 2a5f004edddd456ea83de97d5e73621e (127.0.98.1:34793), c88cbd4878aa4840a35ca9a3b9ea6667 (127.0.98.2:39021)
I20260924 05:11:13.847186 13274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e905371a8840fe97384c7e51d529e2" candidate_uuid: "20eed2d42cd5409389df19c3f31cc733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a5f004edddd456ea83de97d5e73621e"
I20260924 05:11:13.847280 13274 raft_consensus.cc:3037] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:13.848161 13274 raft_consensus.cc:2454] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20eed2d42cd5409389df19c3f31cc733 in term 1.
I20260924 05:11:13.848340 13477 leader_election.cc:302] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20eed2d42cd5409389df19c3f31cc733, 2a5f004edddd456ea83de97d5e73621e; no voters:
I20260924 05:11:13.848490 13629 raft_consensus.cc:2781] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:13.848624 13629 raft_consensus.cc:686] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 1 LEADER]: Becoming Leader. State: Replica: 20eed2d42cd5409389df19c3f31cc733, State: Running, Role: LEADER
I20260924 05:11:13.848726 13629 consensus_queue.cc:237] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } }
I20260924 05:11:13.849593 13122 catalog_manager.cc:5495] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20eed2d42cd5409389df19c3f31cc733 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "20eed2d42cd5409389df19c3f31cc733" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20eed2d42cd5409389df19c3f31cc733" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34585 } health_report { overall_health: HEALTHY } } }
I20260924 05:11:13.852468 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.839947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37177 (local address 127.0.98.2:39021)
0924 05:11:13.839989 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:13.839990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:13.841787 (+ 1797us) server_negotiation.cc:407] Connection header received
0924 05:11:13.842209 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:13.842210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:13.842222 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:13.842251 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:13.842445 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.843122 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.850967 (+ 7845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.851442 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.851485 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.851494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:13.852354 (+ 860us) server_negotiation.cc:299] Negotiation successful
0924 05:11:13.852405 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:11:13.853255 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.839689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39021 (local address 127.0.98.3:37177)
0924 05:11:13.841727 (+ 2038us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:13.841737 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:13.842122 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:13.842307 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:13.842308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:13.842309 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:13.842368 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.842370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.843476 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.843508 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.850867 (+ 7359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:13.850870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.852004 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.852005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:13.852074 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.852079 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:11:13.853126 (+ 1047us) client_negotiation.cc:240] Negotiation successful
0924 05:11:13.853170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2020,"thread_start_us":57,"threads_started":1}
I20260924 05:11:13.853639 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e905371a8840fe97384c7e51d529e2" candidate_uuid: "20eed2d42cd5409389df19c3f31cc733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" is_pre_election: true
I20260924 05:11:13.853803 13408 raft_consensus.cc:2454] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20eed2d42cd5409389df19c3f31cc733 in term 0.
I20260924 05:11:13.853956 13407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e905371a8840fe97384c7e51d529e2" candidate_uuid: "20eed2d42cd5409389df19c3f31cc733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667"
I20260924 05:11:13.854025 13407 raft_consensus.cc:3037] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:13.854672 13407 raft_consensus.cc:2454] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20eed2d42cd5409389df19c3f31cc733 in term 1.
I20260924 05:11:13.871050 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.867761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.0.98.3:34585)
0924 05:11:13.867797 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:13.867798 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:13.867805 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:11:13.867837 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:13.867837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:13.867857 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:13.867915 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:11:13.868081 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.868796 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.869973 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:13.870266 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:13.870323 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:13.870504 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:11:13.870622 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:11:13.870701 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:13.870947 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 05:11:13.870979 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:11:13.875012 13274 raft_consensus.cc:1261] T 25e905371a8840fe97384c7e51d529e2 P 2a5f004edddd456ea83de97d5e73621e [term 1 FOLLOWER]: Refusing update from remote peer 20eed2d42cd5409389df19c3f31cc733: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:13.875308 13629 consensus_queue.cc:1035] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a5f004edddd456ea83de97d5e73621e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:13.877584 13407 raft_consensus.cc:1261] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [term 1 FOLLOWER]: Refusing update from remote peer 20eed2d42cd5409389df19c3f31cc733: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 05:11:13.879622 13589 tablet.cc:2369] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:11:13.880116 13634 consensus_queue.cc:1035] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c88cbd4878aa4840a35ca9a3b9ea6667" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260924 05:11:13.884650 13639 mvcc.cc:205] Tried to move back new op lower bound from 7332768456188387328 to 7332768456084774912. Current Snapshot: MvccSnapshot[applied={T|T < 7332768456188387328}]
I20260924 05:11:13.885193 13638 mvcc.cc:205] Tried to move back new op lower bound from 7332768456188387328 to 7332768456084774912. Current Snapshot: MvccSnapshot[applied={T|T < 7332768456188387328}]
I20260924 05:11:13.888948 13121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:13.889031 13121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:13.889853 13121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5b92226a2d5497fb9f4a7a650b64042: Sending DeleteTablet for 3 replicas of tablet 25e905371a8840fe97384c7e51d529e2
I20260924 05:11:13.890173 13516 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e905371a8840fe97384c7e51d529e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46164
I20260924 05:11:13.890942 13647 tablet_replica.cc:330] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: stopping tablet replica
I20260924 05:11:13.891080 13647 raft_consensus.cc:2229] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:11:13.891242 13647 raft_consensus.cc:2258] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:13.891706 13647 ts_tablet_manager.cc:1905] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:13.892856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 13160
I20260924 05:11:13.893376 13388 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e905371a8840fe97384c7e51d529e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48368
I20260924 05:11:13.894858 13648 tablet_replica.cc:330] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: stopping tablet replica
I20260924 05:11:13.894994 13647 ts_tablet_manager.cc:1918] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:13.895061 13647 log.cc:1196] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/25e905371a8840fe97384c7e51d529e2
I20260924 05:11:13.895303 13647 ts_tablet_manager.cc:1939] T 25e905371a8840fe97384c7e51d529e2 P 20eed2d42cd5409389df19c3f31cc733: Deleting consensus metadata
I20260924 05:11:13.895723 13648 raft_consensus.cc:2229] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:13.895852 13648 raft_consensus.cc:2258] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:13.895857 13106 catalog_manager.cc:4841] TS 20eed2d42cd5409389df19c3f31cc733 (127.0.98.3:34585): tablet 25e905371a8840fe97384c7e51d529e2 (table test-table [id=fd658a71f94d4ab2a2f7ba11a02505ba]) successfully deleted
I20260924 05:11:13.896266 13648 ts_tablet_manager.cc:1905] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:13.897898 13648 ts_tablet_manager.cc:1918] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:13.897958 13648 log.cc:1196] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/25e905371a8840fe97384c7e51d529e2
I20260924 05:11:13.898185 13648 ts_tablet_manager.cc:1939] T 25e905371a8840fe97384c7e51d529e2 P c88cbd4878aa4840a35ca9a3b9ea6667: Deleting consensus metadata
I20260924 05:11:13.898530 13108 catalog_manager.cc:4841] TS c88cbd4878aa4840a35ca9a3b9ea6667 (127.0.98.2:39021): tablet 25e905371a8840fe97384c7e51d529e2 (table test-table [id=fd658a71f94d4ab2a2f7ba11a02505ba]) successfully deleted
I20260924 05:11:13.898573 13516 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e905371a8840fe97384c7e51d529e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46164
W20260924 05:11:13.899008 13106 catalog_manager.cc:4808] TS 20eed2d42cd5409389df19c3f31cc733 (127.0.98.3:34585): delete failed for tablet 25e905371a8840fe97384c7e51d529e2 because the tablet was not found. No further retry: Not found: Tablet not found: 25e905371a8840fe97384c7e51d529e2
W20260924 05:11:13.902930 13109 connection.cc:535] client connection to 127.0.98.1:34793 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 05:11:13.903018 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:11:13.903591 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:13.903473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34793 (local address 127.0.0.1:49426)
0924 05:11:13.903521 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:13.903556 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:11:13.903699 13109 catalog_manager.cc:4584] TS 2a5f004edddd456ea83de97d5e73621e (127.0.98.1:34793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25e905371a8840fe97384c7e51d529e2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34793: connect: Connection refused (error 111)
I20260924 05:11:13.903909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 13324
I20260924 05:11:13.910561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 13457
I20260924 05:11:13.918483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 13089
I20260924 05:11:13.928407 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 05:11:14.263652 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 05:11:14.263959 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 05:11:14.607977 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 05:11:14.608 UTC [12156] LOG: received smart shutdown request
2026-09-24 05:11:14.611 UTC [12156] LOG: background worker "logical replication launcher" (PID 12165) exited with exit code 1
2026-09-24 05:11:14.611 UTC [12160] LOG: shutting down
2026-09-24 05:11:14.759 UTC [12156] LOG: database system is shut down
2026-09-24T05:11:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45607 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:11:14 dist-test-slave-mr2f krb5kdc[13658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:11:14 dist-test-slave-mr2f krb5kdc[13658](info): set up 2 sockets
Sep 24 05:11:14 dist-test-slave-mr2f krb5kdc[13658](info): commencing operation
krb5kdc: starting...
W20260924 05:11:15.861063 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:11:15 dist-test-slave-mr2f krb5kdc[13658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:11:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:11:15.973649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44805
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44805,127.0.98.61:36571,127.0.98.60:36987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:16.050693 13671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:16.050925 13671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:16.050963 13671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:16.052028 13671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:16.052098 13671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:16.052121 13671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:16.052150 13671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:16.052186 13671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:16.053622 13671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44805,127.0.98.61:36571,127.0.98.60:36987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:16.053869 13671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:16.054059 13671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:16.058516 13676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:16.058537 13679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:16.058516 13677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:16.058648 13671 server_base.cc:1034] running on GCE node
I20260924 05:11:16.058970 13671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:16.059248 13671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:16.060403 13671 hybrid_clock.cc:648] HybridClock initialized: now 1790226676060389 us; error 45 us; skew 500 ppm
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:16.069212 13671 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:11:16.069566 13671 webserver.cc:460] Webserver started at http://127.0.98.62:36529/ using document root <none> and password file <none>
I20260924 05:11:16.069757 13671 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:16.069796 13671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:16.069901 13671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:16.070869 13671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "617727d97b3642698d11533739ea3b7c"
format_stamp: "Formatted at 2026-09-24 05:11:16 on dist-test-slave-mr2f"
server_key: "3d45bca0b3df58481d7d47c772b08e3b"
server_key_iv: "0d7489e1d5658d8c737baf48ae5cc226"
server_key_version: "encryptionkey@0"
I20260924 05:11:16.071162 13671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "617727d97b3642698d11533739ea3b7c"
format_stamp: "Formatted at 2026-09-24 05:11:16 on dist-test-slave-mr2f"
server_key: "3d45bca0b3df58481d7d47c772b08e3b"
server_key_iv: "0d7489e1d5658d8c737baf48ae5cc226"
server_key_version: "encryptionkey@0"
I20260924 05:11:16.072312 13671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:16.072921 13686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:16.073078 13671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:16.073173 13671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "617727d97b3642698d11533739ea3b7c"
format_stamp: "Formatted at 2026-09-24 05:11:16 on dist-test-slave-mr2f"
server_key: "3d45bca0b3df58481d7d47c772b08e3b"
server_key_iv: "0d7489e1d5658d8c737baf48ae5cc226"
server_key_version: "encryptionkey@0"
I20260924 05:11:16.073226 13671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:16.094349 13671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:16.096705 13671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:16.096825 13671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:16.100665 13671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44805
I20260924 05:11:16.100735 13738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44805 every 8 connection(s)
I20260924 05:11:16.101082 13671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:11:16.101938 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:16.103634 13739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:11:16.108278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 13671
I20260924 05:11:16.108392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:11:16.108596 392 external_mini_cluster.cc:1428] Setting key 176f968a99f5726237576ded589aa411
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:16.113641 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.104342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.0.98.62:44805)
0924 05:11:16.104512 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.104521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.104652 (+ 131us) server_negotiation.cc:407] Connection header received
0924 05:11:16.105127 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.105129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.105145 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.105221 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.105712 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.106503 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.107432 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.107736 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.107816 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.110097 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.110104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.110104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.110110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.111976 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.112338 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.112342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.112343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.112451 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.112662 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.112663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.112663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.112969 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.113058 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.113271 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.113459 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260924 05:11:16.113643 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.104342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44805 (local address 127.0.0.1:42516)
0924 05:11:16.104540 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.104571 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:16.105008 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:16.105248 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:16.105250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:16.105475 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:16.105625 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.105628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.106597 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.106598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.107361 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.107363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.107810 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.107811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.107864 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.108315 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:16.108326 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:16.109996 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:16.112063 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.112069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.112070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.112274 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.112523 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.112524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.112524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.112599 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.113045 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:16.113049 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:16.113101 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 05:11:16.113204 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 05:11:16.113375 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260924 05:11:16.114784 13739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:11:16.115144 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.115014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36571 (local address 127.0.0.1:54704)
0924 05:11:16.115063 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.115111 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 05:11:16.115288 13687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
I20260924 05:11:16.115758 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.115644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36571 (local address 127.0.0.1:54714)
0924 05:11:16.115706 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.115733 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 05:11:16.115892 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36571: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 05:11:16.131584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36571
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44805,127.0.98.61:36571,127.0.98.60:36987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:16.165112 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } attempt: 1
I20260924 05:11:16.165720 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.165552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36571 (local address 127.0.0.1:54718)
0924 05:11:16.165615 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.165665 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 05:11:16.166126 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.165998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36571 (local address 127.0.0.1:54724)
0924 05:11:16.166052 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.166093 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 05:11:16.166254 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36571: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
W20260924 05:11:16.211134 13746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:16.211338 13746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:16.211359 13746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:16.212510 13746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:16.212551 13746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:16.212565 13746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:16.212579 13746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:16.212590 13746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:16.214077 13746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44805,127.0.98.61:36571,127.0.98.60:36987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:16.214308 13746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:16.214537 13746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:16.219290 13754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:16.219295 13752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:16.219307 13751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:16.219486 13746 server_base.cc:1034] running on GCE node
I20260924 05:11:16.219827 13746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:16.220130 13746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:16.221310 13746 hybrid_clock.cc:648] HybridClock initialized: now 1790226676221302 us; error 41 us; skew 500 ppm
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:16.223737 13746 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 05:11:16.224107 13746 webserver.cc:460] Webserver started at http://127.0.98.61:40447/ using document root <none> and password file <none>
I20260924 05:11:16.224295 13746 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:16.224349 13746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:16.224435 13746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:16.225417 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/data/instance:
uuid: "ca33bf73e76d44ba831eab3d163217da"
format_stamp: "Formatted at 2026-09-24 05:11:16 on dist-test-slave-mr2f"
server_key: "39b0612b67a678f447153c27e10e81df"
server_key_iv: "9890385faff3a97f08bb77e92ab6c219"
server_key_version: "encryptionkey@0"
I20260924 05:11:16.225703 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/wal/instance:
uuid: "ca33bf73e76d44ba831eab3d163217da"
format_stamp: "Formatted at 2026-09-24 05:11:16 on dist-test-slave-mr2f"
server_key: "39b0612b67a678f447153c27e10e81df"
server_key_iv: "9890385faff3a97f08bb77e92ab6c219"
server_key_version: "encryptionkey@0"
I20260924 05:11:16.227032 13746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:11:16.227694 13761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:16.227900 13746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:16.227984 13746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/wal
uuid: "ca33bf73e76d44ba831eab3d163217da"
format_stamp: "Formatted at 2026-09-24 05:11:16 on dist-test-slave-mr2f"
server_key: "39b0612b67a678f447153c27e10e81df"
server_key_iv: "9890385faff3a97f08bb77e92ab6c219"
server_key_version: "encryptionkey@0"
I20260924 05:11:16.228037 13746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:16.234449 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } attempt: 2
I20260924 05:11:16.234843 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.234719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36571 (local address 127.0.0.1:54738)
0924 05:11:16.234768 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.234814 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:11:16.235081 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.235025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36571 (local address 127.0.0.1:54740)
0924 05:11:16.235049 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.235066 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 05:11:16.235149 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36571: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36571: connect: Connection refused (error 111)
I20260924 05:11:16.237205 13746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:16.239660 13746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:16.239821 13746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:16.244594 13746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36571
I20260924 05:11:16.244622 13813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36571 every 8 connection(s)
I20260924 05:11:16.245010 13746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:11:16.245843 13814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:16.246200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 13746
I20260924 05:11:16.246311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-1/wal/instance
I20260924 05:11:16.246500 392 external_mini_cluster.cc:1428] Setting key 139a4b014d8c52de6d3f160dcb24abf5
I20260924 05:11:16.247650 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 05:11:16.257244 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.248394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44805 (local address 127.0.0.1:42524)
0924 05:11:16.248558 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.248597 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:16.249026 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:16.249272 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:16.249275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:16.249488 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:16.249723 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.249728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.250595 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.250596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.251423 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.251425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.251841 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.251842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.251966 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.252440 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:16.252447 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:16.254051 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:16.255821 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.255826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.255828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.256068 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.256315 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.256316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.256316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.256462 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.256787 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:16.256792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:16.256901 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 05:11:16.256982 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 05:11:16.257094 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260924 05:11:16.257256 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.248458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.0.98.62:44805)
0924 05:11:16.248521 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.248524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.248671 (+ 147us) server_negotiation.cc:407] Connection header received
0924 05:11:16.249119 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.249121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.249135 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.249206 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.249838 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.250518 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.251513 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.251727 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.251833 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.254144 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.254145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.254145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.254152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.255744 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.256177 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.256177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.256178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.256227 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.256545 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.256546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.256547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.256719 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.256791 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.257026 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.257119 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:11:16.258220 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 05:11:16.268867 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.258508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.0.98.61:36571)
0924 05:11:16.258639 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.258644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.258662 (+ 18us) server_negotiation.cc:407] Connection header received
0924 05:11:16.258682 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.258684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.258695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.262843 (+ 4148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.262886 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.263504 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.264221 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.264515 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.264598 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.266229 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.266235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.266236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.266242 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.267740 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.268115 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.268117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.268117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.268167 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.268413 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.268414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.268415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.268545 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.268612 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.268731 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.268797 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":84,"threads_started":1}
I20260924 05:11:16.269053 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.258472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36571 (local address 127.0.0.1:54750)
0924 05:11:16.258527 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.258540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:16.258586 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:16.258816 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:16.258817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:16.258989 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:16.259035 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.259037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.263574 (+ 4537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.263575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.264109 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.264111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.264620 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.264621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.264690 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.265061 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:16.265065 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:16.266134 (+ 1069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:16.267831 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.267831 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.267832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.268023 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.268231 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.268232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.268232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.268319 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.268605 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:16.268605 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:16.268638 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 05:11:16.268689 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:11:16.268858 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":48}
I20260924 05:11:16.269541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36987
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44805,127.0.98.61:36571,127.0.98.60:36987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:16.269624 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:11:16.269928 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.269816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39280)
0924 05:11:16.269852 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.269898 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 05:11:16.270071 13765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
I20260924 05:11:16.270453 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.270364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39288)
0924 05:11:16.270411 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.270433 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 05:11:16.270644 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36987: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
I20260924 05:11:16.319895 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } attempt: 1
I20260924 05:11:16.320427 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.320284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39302)
0924 05:11:16.320355 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.320396 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 05:11:16.320777 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.320674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39308)
0924 05:11:16.320727 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.320751 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 05:11:16.320871 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36987: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
I20260924 05:11:16.326294 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } attempt: 3
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:11:16.333593 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.326644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.0.98.61:36571)
0924 05:11:16.326712 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.326714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.326724 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:11:16.326793 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.326794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.326804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.326860 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.327181 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.327712 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.328409 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.328629 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.328721 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.330939 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.330941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.330941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.330947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.332409 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.332835 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.332836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.332837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.332895 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.333142 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.333143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.333143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.333249 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.333301 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.333469 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.333533 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 05:11:16.333670 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.326575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36571 (local address 127.0.0.1:54756)
0924 05:11:16.326626 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.326637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:16.326710 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:16.326867 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:16.326868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:16.327028 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:16.327091 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.327092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.327805 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.327806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.328323 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.328325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.328721 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.328722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.328800 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.329259 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:16.329263 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:16.330853 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:16.332488 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.332489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.332490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.332755 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.332997 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.332997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.332998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.333070 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.333318 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:16.333319 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:16.333371 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 05:11:16.333446 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:11:16.333550 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:11:16.334074 13739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:11:16.334421 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.334307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39316)
0924 05:11:16.334359 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.334396 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:11:16.334746 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.334652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39330)
0924 05:11:16.334697 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.334725 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 05:11:16.334888 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36987: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
W20260924 05:11:16.349517 13821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:16.349705 13821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:16.349726 13821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:16.350816 13821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:16.350862 13821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:16.350878 13821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:16.350890 13821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:16.350901 13821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:16.352288 13821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44805,127.0.98.61:36571,127.0.98.60:36987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:16.352478 13821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:16.352648 13821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:16.357303 13829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:16.357348 13821 server_base.cc:1034] running on GCE node
W20260924 05:11:16.357333 13827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:16.357317 13826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:16.357738 13821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:16.358048 13821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:16.359208 13821 hybrid_clock.cc:648] HybridClock initialized: now 1790226676359203 us; error 37 us; skew 500 ppm
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:16.361524 13821 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 05:11:16.361982 13821 webserver.cc:460] Webserver started at http://127.0.98.60:33041/ using document root <none> and password file <none>
I20260924 05:11:16.362197 13821 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:16.362264 13821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:16.362409 13821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:16.363472 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/data/instance:
uuid: "d96d19313dfd4e80a01c19ad715fd29d"
format_stamp: "Formatted at 2026-09-24 05:11:16 on dist-test-slave-mr2f"
server_key: "00a1e2a9ea6eea4e6e67c66722389621"
server_key_iv: "a324a8b9acfc04c64cced20e78956c21"
server_key_version: "encryptionkey@0"
I20260924 05:11:16.363817 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/wal/instance:
uuid: "d96d19313dfd4e80a01c19ad715fd29d"
format_stamp: "Formatted at 2026-09-24 05:11:16 on dist-test-slave-mr2f"
server_key: "00a1e2a9ea6eea4e6e67c66722389621"
server_key_iv: "a324a8b9acfc04c64cced20e78956c21"
server_key_version: "encryptionkey@0"
I20260924 05:11:16.365096 13821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:11:16.365752 13836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:16.365914 13821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:16.366029 13821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/wal
uuid: "d96d19313dfd4e80a01c19ad715fd29d"
format_stamp: "Formatted at 2026-09-24 05:11:16 on dist-test-slave-mr2f"
server_key: "00a1e2a9ea6eea4e6e67c66722389621"
server_key_iv: "a324a8b9acfc04c64cced20e78956c21"
server_key_version: "encryptionkey@0"
I20260924 05:11:16.366029 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } attempt: 1
I20260924 05:11:16.366129 13821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:16.366474 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.366349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39344)
0924 05:11:16.366392 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.366443 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:11:16.366818 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.366708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39358)
0924 05:11:16.366752 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.366792 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:11:16.366904 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36987: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36987: connect: Connection refused (error 111)
I20260924 05:11:16.380674 13821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:16.382412 13821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:16.382579 13821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:16.386382 13821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36987
I20260924 05:11:16.386413 13888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36987 every 8 connection(s)
I20260924 05:11:16.386749 13821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:11:16.387634 13889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:16.389021 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } attempt: 2
I20260924 05:11:16.389194 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:11:16.393601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 13821
I20260924 05:11:16.393725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790226527740525-392-0/minicluster-data/master-2/wal/instance
I20260924 05:11:16.393929 392 external_mini_cluster.cc:1428] Setting key 2a8bc883c044c064444dec4d0812bc0b
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:16.398497 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.389294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39368)
0924 05:11:16.389367 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.389380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:16.389429 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:16.390322 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:16.390324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:16.390504 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:16.390545 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.390547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.391605 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.391607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.392106 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.392108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.392608 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.392610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.392662 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.393202 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:16.393208 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:16.395074 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:16.397124 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.397126 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.397126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.397338 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.397623 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.397624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.397624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.397694 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.398111 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:16.398112 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:16.398188 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 05:11:16.398266 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 05:11:16.398382 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 05:11:16.398660 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.389937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.0.98.62:44805)
0924 05:11:16.390008 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.390010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.390190 (+ 180us) server_negotiation.cc:407] Connection header received
0924 05:11:16.390271 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.390272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.390283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.390329 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.390881 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.391357 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.392266 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.392459 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.392542 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.395492 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.395494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.395494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.395501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.396974 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.397434 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.397435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.397435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.397500 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.397757 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.397758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.397758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.397900 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.398215 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.398469 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.398553 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 05:11:16.399036 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.389468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.0.98.60:36987)
0924 05:11:16.389648 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.389657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.389692 (+ 35us) server_negotiation.cc:407] Connection header received
0924 05:11:16.390147 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.390154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.390202 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.390326 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.390637 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.391503 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.392213 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.392527 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.392643 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.395156 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.395164 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.395164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.395173 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.397015 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.397435 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.397438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.397438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.397519 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.397770 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.397771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.397772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.398034 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.398113 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.398344 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.398476 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260924 05:11:16.399036 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.389881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44805 (local address 127.0.0.1:42534)
0924 05:11:16.390067 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.390101 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:16.390211 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:16.390358 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:16.390360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:16.390618 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:16.390805 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.390811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.391452 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.391453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.392161 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.392163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.392551 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.392552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.392643 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.393297 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:16.393305 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:16.395358 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:16.397080 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.397088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.397090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.397364 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.397574 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.397574 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.397575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.397669 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.398280 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:16.398286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:16.398339 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 05:11:16.398732 (+ 393us) client_negotiation.cc:240] Negotiation successful
0924 05:11:16.398847 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:11:16.400233 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:11:16.400841 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da: Bootstrap starting.
I20260924 05:11:16.401952 13814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:16.402254 13814 log.cc:824] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:16.403151 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da: No bootstrap required, opened a new log
I20260924 05:11:16.404266 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.394304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.0.98.60:36987)
0924 05:11:16.394567 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.394568 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.394573 (+ 5us) server_negotiation.cc:407] Connection header received
0924 05:11:16.394586 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.394587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.394593 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.394639 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.395442 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.396232 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.397317 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.397623 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.397715 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.400409 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.400411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.400411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.400418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.402938 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.403305 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.403306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.403306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.403360 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.403576 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.403576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.403577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.403703 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.404036 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.404074 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.404147 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":72,"threads_started":1}
I20260924 05:11:16.405694 13814 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } }
I20260924 05:11:16.405838 13814 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:16.405886 13814 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca33bf73e76d44ba831eab3d163217da, State: Initialized, Role: FOLLOWER
I20260924 05:11:16.406001 13814 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } }
I20260924 05:11:16.406322 13814 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:11:16.407899 13896 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.407972 13896 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:11:16.408974 13907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:11:16.409070 13907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:16.410671 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.400488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36571 (local address 127.0.0.1:54760)
0924 05:11:16.400543 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.400555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:16.400599 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:16.402818 (+ 2219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:16.402820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:16.402996 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:16.403034 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.403036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.403885 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.403886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.404610 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.404612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.405044 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.405045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.405101 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.406217 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:16.406222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:16.407708 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:16.409519 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.409520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.409520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.409710 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.409912 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.409913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.409914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.409975 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.410310 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:16.410311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:16.410357 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:11:16.410420 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:11:16.410538 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 05:11:16.411067 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.402556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.0.98.61:36571)
0924 05:11:16.402610 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.402612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.402620 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:11:16.402642 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.402643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.402653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.402723 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.403125 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.403805 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.404701 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.404985 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.405039 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.407799 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.407801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.407801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.407809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.409426 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.409792 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.409794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.409794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.409845 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.410053 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.410054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.410054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.410226 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.410283 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.410900 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.410979 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:11:16.411393 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:11:16.408529 13746 master_runner.cc:186] Error getting master registration for 127.0.98.62:44805: OK, instance_id { permanent_uuid: "617727d97b3642698d11533739ea3b7c" instance_seqno: 1790226676099561 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:11:16.411506 13746 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:11:16.413354 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.404774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.0.98.62:44805)
0924 05:11:16.404807 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.404808 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.404815 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:11:16.404880 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.404881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.404890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.404941 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.405528 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.406098 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.407103 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.407310 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.407403 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.410206 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.410207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.410208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.410214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.411883 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.412255 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.412256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.412256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.412309 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.412521 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.412522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.412522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.412631 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.413210 (+ 579us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.413242 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.413299 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:11:16.419044 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.411628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39388)
0924 05:11:16.411686 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.411695 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:16.411730 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:16.411873 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:16.411874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:16.412015 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:16.412050 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.412052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.412909 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.412910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.413615 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.413617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.414083 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.414084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.414142 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.414521 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:16.414526 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:16.415947 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:16.417944 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.417945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.417946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.418143 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.418370 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.418371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.418371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.418430 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.418781 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:16.418781 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:16.418818 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 05:11:16.418866 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 05:11:16.418955 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 05:11:16.419327 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.411675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.0.98.60:36987)
0924 05:11:16.411709 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.411710 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.411721 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:11:16.411779 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.411780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.411791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.411869 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.412129 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.412801 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.413697 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.413990 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.414078 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.416034 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.416036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.416036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.416043 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.417840 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.418232 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.418233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.418233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.418295 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.418503 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.418504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.418504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.418669 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.419158 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.419213 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.419281 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:11:16.420835 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d: Bootstrap starting.
I20260924 05:11:16.421530 13889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:16.421836 13889 log.cc:824] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:16.422295 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.413890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.0.98.61:36571)
0924 05:11:16.413923 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.413925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.413964 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:11:16.414030 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.414031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.414039 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.414089 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.414840 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.415369 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.416199 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.416489 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.416561 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.418963 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.418964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.418965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.418972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.420975 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.421368 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.421370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.421370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.421430 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.421687 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.421689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.421689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.421859 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.422094 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.422141 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.422211 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 05:11:16.422550 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d: No bootstrap required, opened a new log
I20260924 05:11:16.423986 13889 raft_consensus.cc:348] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } }
I20260924 05:11:16.424100 13889 raft_consensus.cc:374] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:16.424120 13889 raft_consensus.cc:729] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96d19313dfd4e80a01c19ad715fd29d, State: Initialized, Role: FOLLOWER
I20260924 05:11:16.424235 13889 consensus_queue.cc:260] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } }
I20260924 05:11:16.424513 13889 sys_catalog.cc:564] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:16.424490 13910 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.424621 13910 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:11:16.426508 13921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:11:16.426573 13921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:11:16.426913 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:11:16.426820 13821 master_runner.cc:186] Error getting master registration for 127.0.98.62:44805: OK, instance_id { permanent_uuid: "617727d97b3642698d11533739ea3b7c" instance_seqno: 1790226676099561 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:11:16.429253 13821 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:11:16.439798 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.428634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.0.98.60:36987)
0924 05:11:16.428665 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.428667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.431160 (+ 2493us) server_negotiation.cc:407] Connection header received
0924 05:11:16.431254 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.431255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.431264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.431309 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.431897 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.432642 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.434519 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.434855 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.434898 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.436673 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.436674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.436675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.436691 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.438365 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.438804 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.438806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.438806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.438865 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.439111 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.439112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.439112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.439269 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.439362 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.439590 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.439662 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:11:16.442374 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.428902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.0.98.61:36571)
0924 05:11:16.428945 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.428947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.432951 (+ 4004us) server_negotiation.cc:407] Connection header received
0924 05:11:16.433038 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.433039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.433049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.433077 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.433740 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.434433 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.435630 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.435918 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.435963 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.438190 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.438191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.438191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.438199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.439973 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.440952 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.440953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.440953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.441012 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.441510 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.441512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.441512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.441646 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.441719 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.442192 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.442269 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:11:16.442708 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } attempt: 2
I20260924 05:11:16.444000 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.428345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.0.98.62:44805)
0924 05:11:16.428383 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.428385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.430986 (+ 2601us) server_negotiation.cc:407] Connection header received
0924 05:11:16.431085 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.431086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.431097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.431148 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.431948 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.432647 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.435228 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.435515 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.435553 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.440960 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.440961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.440961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.440968 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.442621 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.443155 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.443156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.443157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.443213 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.443418 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.443419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.443419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.443546 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.443639 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.443849 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.443918 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 05:11:16 dist-test-slave-mr2f krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:11:16.449554 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.443286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.0.98.60:36987)
0924 05:11:16.443425 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:16.443427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:16.443434 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:11:16.443451 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:16.443452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:16.443461 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:16.443503 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:16.443803 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.444425 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.445155 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.445347 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.445415 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.447129 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:16.447130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:16.447131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:16.447137 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:16.448431 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.448790 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.448791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.448791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.448847 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:16.449079 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:16.449079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:16.449080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:16.449181 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:16.449251 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:16.449410 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 05:11:16.449472 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260924 05:11:16.449554 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:16.443078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36987 (local address 127.0.0.1:39408)
0924 05:11:16.443161 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:16.443170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:16.443217 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:16.443540 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:16.443541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:16.443687 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:16.443724 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.443726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.444511 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.444512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.445073 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:16.445076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:16.445423 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:16.445424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:16.445480 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:16.445879 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:16.445883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:16.447036 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:16.448546 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.448547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.448547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.448716 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.448949 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:16.448950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:16.448951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:16.449006 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:16.449246 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:16.449247 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:16.449288 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 05:11:16.449372 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:11:16.449463 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 05:11:16.451396 13739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c: Bootstrap starting.
I20260924 05:11:16.452039 13739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:16.452463 13739 log.cc:824] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:16.453473 13739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c: No bootstrap required, opened a new log
I20260924 05:11:16.455022 13739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } }
I20260924 05:11:16.455158 13739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:16.455217 13739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617727d97b3642698d11533739ea3b7c, State: Initialized, Role: FOLLOWER
I20260924 05:11:16.455344 13739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } }
I20260924 05:11:16.455629 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.455684 13739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:16.455757 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:11:16.458297 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:11:16.458401 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:11:16.459848 13671 master_runner.cc:263] No leader master found from master 617727d97b3642698d11533739ea3b7c
I20260924 05:11:16.459894 13671 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 05:11:16.490414 13767 tablet.cc:2369] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:11:16.525357 13896 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:16.525490 13896 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } }
I20260924 05:11:16.525731 13896 leader_election.cc:288] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 617727d97b3642698d11533739ea3b7c (127.0.98.62:44805), d96d19313dfd4e80a01c19ad715fd29d (127.0.98.60:36987)
I20260924 05:11:16.526206 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca33bf73e76d44ba831eab3d163217da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617727d97b3642698d11533739ea3b7c" is_pre_election: true
I20260924 05:11:16.526336 13713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca33bf73e76d44ba831eab3d163217da in term 0.
I20260924 05:11:16.526507 13765 leader_election.cc:302] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 617727d97b3642698d11533739ea3b7c, ca33bf73e76d44ba831eab3d163217da; no voters:
I20260924 05:11:16.526922 13863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca33bf73e76d44ba831eab3d163217da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d96d19313dfd4e80a01c19ad715fd29d" is_pre_election: true
I20260924 05:11:16.527096 13863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca33bf73e76d44ba831eab3d163217da in term 0.
I20260924 05:11:16.527472 13896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:11:16.527529 13896 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:11:16.527554 13896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:16.528548 13896 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } }
I20260924 05:11:16.528656 13896 leader_election.cc:288] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [CANDIDATE]: Term 1 election: Requested vote from peers 617727d97b3642698d11533739ea3b7c (127.0.98.62:44805), d96d19313dfd4e80a01c19ad715fd29d (127.0.98.60:36987)
I20260924 05:11:16.529044 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca33bf73e76d44ba831eab3d163217da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617727d97b3642698d11533739ea3b7c"
I20260924 05:11:16.529132 13713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:16.530009 13713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca33bf73e76d44ba831eab3d163217da in term 1.
I20260924 05:11:16.530239 13765 leader_election.cc:302] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 617727d97b3642698d11533739ea3b7c, ca33bf73e76d44ba831eab3d163217da; no voters:
I20260924 05:11:16.530412 13863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca33bf73e76d44ba831eab3d163217da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d96d19313dfd4e80a01c19ad715fd29d"
I20260924 05:11:16.530501 13863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:16.531472 13863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca33bf73e76d44ba831eab3d163217da in term 1.
I20260924 05:11:16.531713 13896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:16.531883 13896 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [term 1 LEADER]: Becoming Leader. State: Replica: ca33bf73e76d44ba831eab3d163217da, State: Running, Role: LEADER
I20260924 05:11:16.531970 13896 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } }
I20260924 05:11:16.532318 13896 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca33bf73e76d44ba831eab3d163217da. Latest consensus state: current_term: 1 leader_uuid: "ca33bf73e76d44ba831eab3d163217da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.532401 13896 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:16.533219 13946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:11:16.533457 13946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:11:16.535753 13863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [term 1 FOLLOWER]: Refusing update from remote peer ca33bf73e76d44ba831eab3d163217da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:16.536103 13944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [LEADER]: Connected to new peer: Peer: permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:16.537439 13910 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca33bf73e76d44ba831eab3d163217da. Latest consensus state: current_term: 1 leader_uuid: "ca33bf73e76d44ba831eab3d163217da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.537541 13910 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:11:16.537904 13713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [term 1 FOLLOWER]: Refusing update from remote peer ca33bf73e76d44ba831eab3d163217da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:16.538210 13944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [LEADER]: Connected to new peer: Peer: permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:16.539449 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca33bf73e76d44ba831eab3d163217da. Latest consensus state: current_term: 1 leader_uuid: "ca33bf73e76d44ba831eab3d163217da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.539528 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:11:16.540437 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca33bf73e76d44ba831eab3d163217da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.540510 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:16.540904 13910 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca33bf73e76d44ba831eab3d163217da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.540974 13910 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:11:16.541198 13947 mvcc.cc:205] Tried to move back new op lower bound from 7332768467088306176 to 7332768467075387392. Current Snapshot: MvccSnapshot[applied={T|T < 7332768467088306176}]
I20260924 05:11:16.541682 13948 mvcc.cc:205] Tried to move back new op lower bound from 7332768467088306176 to 7332768467075387392. Current Snapshot: MvccSnapshot[applied={T|T < 7332768467088306176}]
I20260924 05:11:16.542176 13946 catalog_manager.cc:1347] Generated new cluster ID: 91b6faaf8e9246c195549222e36f780c
I20260924 05:11:16.542218 13946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:11:16.547760 13896 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca33bf73e76d44ba831eab3d163217da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.547851 13896 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:16.548219 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca33bf73e76d44ba831eab3d163217da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617727d97b3642698d11533739ea3b7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } peers { permanent_uuid: "ca33bf73e76d44ba831eab3d163217da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36571 } } peers { permanent_uuid: "d96d19313dfd4e80a01c19ad715fd29d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36987 } } }
I20260924 05:11:16.548297 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:11:16.548378 13950 mvcc.cc:205] Tried to move back new op lower bound from 7332768467088306176 to 7332768467075387392. Current Snapshot: MvccSnapshot[applied={T|T < 7332768467088306176}]
I20260924 05:11:16.571825 13946 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:11:16.572610 13946 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:11:16.581576 13946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca33bf73e76d44ba831eab3d163217da: Generated new TSK 0
I20260924 05:11:16.581821 13946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:11:17.427081 13921 catalog_manager.cc:1259] Loaded cluster ID: 91b6faaf8e9246c195549222e36f780c
I20260924 05:11:17.427160 13921 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d: loading cluster ID for follower catalog manager: success
I20260924 05:11:17.428072 13921 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d: acquiring CA information for follower catalog manager: success
I20260924 05:11:17.428369 13921 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d96d19313dfd4e80a01c19ad715fd29d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:11:17.458927 13942 catalog_manager.cc:1259] Loaded cluster ID: 91b6faaf8e9246c195549222e36f780c
I20260924 05:11:17.458992 13942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c: loading cluster ID for follower catalog manager: success
I20260924 05:11:17.459777 13942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c: acquiring CA information for follower catalog manager: success
I20260924 05:11:17.460017 13942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 617727d97b3642698d11533739ea3b7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:11:17.640089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 13671
I20260924 05:11:17.645274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 13746
I20260924 05:11:17.649951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 13821
2026-09-24T05:11:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2791 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:11:17 dist-test-slave-mr2f krb5kdc[13962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:11:17 dist-test-slave-mr2f krb5kdc[13962](info): set up 2 sockets
Sep 24 05:11:17 dist-test-slave-mr2f krb5kdc[13962](info): commencing operation
krb5kdc: starting...
W20260924 05:11:18.666911 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:11:18 dist-test-slave-mr2f krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:11:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:11:18.777590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39457
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:18.853878 13975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:18.854099 13975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:18.854132 13975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:18.855290 13975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:18.855355 13975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:18.855379 13975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:18.855407 13975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:18.855438 13975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:18.856890 13975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:18.857152 13975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:18.857366 13975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:18.861974 13981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:18.861974 13983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:18.861997 13980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:18.862020 13975 server_base.cc:1034] running on GCE node
I20260924 05:11:18.862365 13975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:18.862641 13975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:18.863783 13975 hybrid_clock.cc:648] HybridClock initialized: now 1790226678863768 us; error 32 us; skew 500 ppm
Sep 24 05:11:18 dist-test-slave-mr2f krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:18.866122 13975 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:11:18.866597 13975 webserver.cc:460] Webserver started at http://127.0.98.62:45257/ using document root <none> and password file <none>
I20260924 05:11:18.866868 13975 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:18.866936 13975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:18.867090 13975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:18.868084 13975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea"
format_stamp: "Formatted at 2026-09-24 05:11:18 on dist-test-slave-mr2f"
server_key: "05b1597cdf3c98aa9e56269b98f65a71"
server_key_iv: "f2cae37744bcbadcd1c644b53290632d"
server_key_version: "encryptionkey@0"
I20260924 05:11:18.868431 13975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea"
format_stamp: "Formatted at 2026-09-24 05:11:18 on dist-test-slave-mr2f"
server_key: "05b1597cdf3c98aa9e56269b98f65a71"
server_key_iv: "f2cae37744bcbadcd1c644b53290632d"
server_key_version: "encryptionkey@0"
I20260924 05:11:18.869727 13975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:11:18.870390 13990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:18.870550 13975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:11:18.870643 13975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea"
format_stamp: "Formatted at 2026-09-24 05:11:18 on dist-test-slave-mr2f"
server_key: "05b1597cdf3c98aa9e56269b98f65a71"
server_key_iv: "f2cae37744bcbadcd1c644b53290632d"
server_key_version: "encryptionkey@0"
I20260924 05:11:18.870721 13975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:18.888005 13975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:18.889662 13975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:18.889818 13975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:18.893436 13975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39457
I20260924 05:11:18.893483 14042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39457 every 8 connection(s)
I20260924 05:11:18.893803 13975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:11:18.894626 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:18.896826 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea: Bootstrap starting.
I20260924 05:11:18.897382 14043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:18.897660 14043 log.cc:824] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:18.898307 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea: No bootstrap required, opened a new log
I20260924 05:11:18.899856 14043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39457 } }
I20260924 05:11:18.899982 14043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:18.900003 14043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ea6cc0c03a4399b4e66ff1d33e36ea, State: Initialized, Role: FOLLOWER
I20260924 05:11:18.900192 14043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39457 } }
I20260924 05:11:18.900269 14043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:11:18.900310 14043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:11:18.900358 14043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:18.900826 14043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39457 } }
I20260924 05:11:18.900933 14043 leader_election.cc:302] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ea6cc0c03a4399b4e66ff1d33e36ea; no voters:
I20260924 05:11:18.901100 14043 leader_election.cc:288] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:11:18.901168 14046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:18.901342 14046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [term 1 LEADER]: Becoming Leader. State: Replica: 67ea6cc0c03a4399b4e66ff1d33e36ea, State: Running, Role: LEADER
I20260924 05:11:18.901350 14043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:18.901448 14046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39457 } }
I20260924 05:11:18.901971 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39457 } } }
I20260924 05:11:18.902045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 13975
I20260924 05:11:18.902057 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:18.902132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:11:18.902328 392 external_mini_cluster.cc:1428] Setting key 2f9b7356f516b280b47c0cb1b2dc705b
I20260924 05:11:18.902460 14048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ea6cc0c03a4399b4e66ff1d33e36ea. Latest consensus state: current_term: 1 leader_uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ea6cc0c03a4399b4e66ff1d33e36ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39457 } } }
I20260924 05:11:18.902541 14048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:18.902653 14052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:11:18.903190 14052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:11:18.904660 14052 catalog_manager.cc:1347] Generated new cluster ID: ef6b25317fc04d92a1bb109719512be8
I20260924 05:11:18.904747 14052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:11:18 dist-test-slave-mr2f krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:18.911859 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:18.902753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.0.98.62:39457)
0924 05:11:18.902935 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:18.902944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:18.903019 (+ 75us) server_negotiation.cc:407] Connection header received
0924 05:11:18.903157 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:18.903164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:18.903213 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:18.903962 (+ 749us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:18.904027 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:18.905084 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:18.905855 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:18.906267 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:18.906424 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:18.908656 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:18.908659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:18.908659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:18.908670 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:18.910362 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:18.910696 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:18.910699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:18.910699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:18.910857 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:18.911062 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:18.911075 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:18.911075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:18.911304 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:18.911492 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:18.911549 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 05:11:18.911669 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260924 05:11:18.913865 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:11:18.914795 14052 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:11:18.915473 14052 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:11:18.927521 14052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67ea6cc0c03a4399b4e66ff1d33e36ea: Generated new TSK 0
I20260924 05:11:18.927707 14052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:11:18.967792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 13975
2026-09-24T05:11:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1316 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:11:19 dist-test-slave-mr2f krb5kdc[14070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:11:19 dist-test-slave-mr2f krb5kdc[14070](info): set up 2 sockets
Sep 24 05:11:19 dist-test-slave-mr2f krb5kdc[14070](info): commencing operation
krb5kdc: starting...
W20260924 05:11:19.954963 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:11:20.065054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:20.140435 14083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:20.140604 14083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:20.140623 14083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:20.141631 14083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:20.141664 14083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:20.141685 14083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:20.141696 14083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:20.141708 14083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:20.143088 14083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:20.143255 14083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:20.143401 14083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:20.147842 14089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:20.147914 14083 server_base.cc:1034] running on GCE node
W20260924 05:11:20.147858 14091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:20.147850 14088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:20.148360 14083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:20.148590 14083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:20.149729 14083 hybrid_clock.cc:648] HybridClock initialized: now 1790226680149714 us; error 33 us; skew 500 ppm
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:20.152254 14083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:11:20.152700 14083 webserver.cc:460] Webserver started at http://127.0.98.62:40727/ using document root <none> and password file <none>
I20260924 05:11:20.152922 14083 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:20.152964 14083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:20.153095 14083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:20.154042 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "327fd3ed012041d8830a50b2e21bc180"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "f0518dc4da732db4a87bf42489b84bee"
server_key_iv: "7c24df4da608a58c7f20a5c09c32c0f3"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.154379 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "327fd3ed012041d8830a50b2e21bc180"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "f0518dc4da732db4a87bf42489b84bee"
server_key_iv: "7c24df4da608a58c7f20a5c09c32c0f3"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.155654 14083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:11:20.156323 14098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.156538 14083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.156621 14083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "327fd3ed012041d8830a50b2e21bc180"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "f0518dc4da732db4a87bf42489b84bee"
server_key_iv: "7c24df4da608a58c7f20a5c09c32c0f3"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.156697 14083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:20.174199 14083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:20.175943 14083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:20.176096 14083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:20.180001 14083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36995
I20260924 05:11:20.180064 14150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36995 every 8 connection(s)
I20260924 05:11:20.180383 14083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:11:20.181283 14151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:20.183459 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Bootstrap starting.
I20260924 05:11:20.184006 14151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:20.184278 14151 log.cc:824] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:20.184924 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: No bootstrap required, opened a new log
I20260924 05:11:20.186455 14151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:20.186604 14151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:20.186643 14151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327fd3ed012041d8830a50b2e21bc180, State: Initialized, Role: FOLLOWER
I20260924 05:11:20.186820 14151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:20.186915 14151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:11:20.186956 14151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:11:20.187009 14151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:20.187765 14151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:20.187896 14151 leader_election.cc:302] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 327fd3ed012041d8830a50b2e21bc180; no voters:
I20260924 05:11:20.188072 14151 leader_election.cc:288] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:11:20.188175 14154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:20.188315 14151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:20.188338 14154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 1 LEADER]: Becoming Leader. State: Replica: 327fd3ed012041d8830a50b2e21bc180, State: Running, Role: LEADER
I20260924 05:11:20.188529 14154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:20.188915 14155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "327fd3ed012041d8830a50b2e21bc180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } } }
I20260924 05:11:20.188930 14156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 327fd3ed012041d8830a50b2e21bc180. Latest consensus state: current_term: 1 leader_uuid: "327fd3ed012041d8830a50b2e21bc180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } } }
I20260924 05:11:20.189013 14155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:20.189047 14156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:20.189241 14163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:11:20.189425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 14083
I20260924 05:11:20.189551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:11:20.189780 392 external_mini_cluster.cc:1428] Setting key da7ba7eef059079e8251de0ea39261c4
I20260924 05:11:20.189779 14163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:11:20.192185 14163 catalog_manager.cc:1347] Generated new cluster ID: 67437826f29449948c96324a318f43c3
I20260924 05:11:20.192273 14163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:20.199549 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.190406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.0.98.62:36995)
0924 05:11:20.190549 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.190559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.190588 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:11:20.190725 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.190732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.190809 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.190977 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:20.191613 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.192833 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.193636 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.194028 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.194146 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.196458 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:20.196462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:20.196462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:20.196473 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:20.198083 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.198447 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.198450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.198450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.198546 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.198751 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.198753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.198753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.198992 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:20.199178 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.199241 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.199378 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260924 05:11:20.200677 14163 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:11:20.201295 14163 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:11:20.209311 14163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Generated new TSK 0
I20260924 05:11:20.209462 14163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:11:20.231518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36995
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:20.311033 14177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:20.311218 14177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:20.311239 14177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:20.312316 14177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:20.312354 14177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:20.312378 14177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:11:20.313848 14177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:20.314055 14177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:20.314281 14177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:20.318996 14183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:20.319085 14177 server_base.cc:1034] running on GCE node
W20260924 05:11:20.318987 14182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:20.319156 14185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:20.319371 14177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:20.319621 14177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:20.320782 14177 hybrid_clock.cc:648] HybridClock initialized: now 1790226680320774 us; error 29 us; skew 500 ppm
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:20.323194 14177 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:11:20.323608 14177 webserver.cc:460] Webserver started at http://127.0.98.1:43913/ using document root <none> and password file <none>
I20260924 05:11:20.323781 14177 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:20.323818 14177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:20.323899 14177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:20.324815 14177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "e35da93437b443ee943b114b6e2245a5"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "98651adbb1d6e5013246c878b526f084"
server_key_iv: "3bfbe7103129eacc7733cdc8a0ac2225"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.325088 14177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e35da93437b443ee943b114b6e2245a5"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "98651adbb1d6e5013246c878b526f084"
server_key_iv: "3bfbe7103129eacc7733cdc8a0ac2225"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.326314 14177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:11:20.327069 14192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.327239 14177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.327343 14177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "e35da93437b443ee943b114b6e2245a5"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "98651adbb1d6e5013246c878b526f084"
server_key_iv: "3bfbe7103129eacc7733cdc8a0ac2225"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.327417 14177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:20.341717 14177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:20.343842 14177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:20.343989 14177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:20.344210 14177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:20.344501 14177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:20.344570 14177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.344609 14177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:20.344635 14177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.350167 14177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40981
I20260924 05:11:20.350214 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40981 every 8 connection(s)
I20260924 05:11:20.350492 14177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:11:20.355733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 14177
I20260924 05:11:20.355846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:20.356077 392 external_mini_cluster.cc:1428] Setting key b24f30f19bfccf2b186ce2529f0cdaae
I20260924 05:11:20.359243 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.351142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56075 (local address 127.0.98.62:36995)
0924 05:11:20.351184 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.351186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.351406 (+ 220us) server_negotiation.cc:407] Connection header received
0924 05:11:20.351820 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.351821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.351834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.351883 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:20.352706 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.353212 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.354091 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.354291 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.354363 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.356542 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:20.356543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:20.356544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:20.356551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:20.357868 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.358240 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.358241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.358241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.358298 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.358564 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.358565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.358565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.358678 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:20.358753 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.359036 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.359129 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:11:20.359357 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.351124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:56075)
0924 05:11:20.351285 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:20.351317 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:20.351746 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:20.351950 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:20.351953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:20.352469 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:20.352636 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.352640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.353282 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.353283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.354024 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.354026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.354396 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.354396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.354507 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.354939 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:20.354947 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:20.356459 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:20.357935 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:20.357942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:20.357944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:20.358161 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:20.358375 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:20.358375 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:20.358376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:20.358486 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:20.358744 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:20.358751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:20.358899 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 05:11:20.359001 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 05:11:20.359147 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260924 05:11:20.359827 14306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36995
I20260924 05:11:20.359939 14306 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:20.360112 14306 heartbeater.cc:511] Master 127.0.98.62:36995 requested a full tablet report, sending...
I20260924 05:11:20.360610 14115 ts_manager.cc:194] Registered new tserver with Master: e35da93437b443ee943b114b6e2245a5 (127.0.98.1:40981)
I20260924 05:11:20.361181 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56075
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:11:20.379560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36995
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:20.454285 14311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:20.454445 14311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:20.454465 14311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:20.455551 14311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:20.455590 14311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:20.455615 14311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:11:20.456954 14311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:20.457130 14311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:20.457298 14311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:20.461885 14316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:20.461895 14319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:20.462023 14311 server_base.cc:1034] running on GCE node
W20260924 05:11:20.461897 14317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:20.462329 14311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:20.462561 14311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:20.463717 14311 hybrid_clock.cc:648] HybridClock initialized: now 1790226680463700 us; error 31 us; skew 500 ppm
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:20.466063 14311 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:11:20.466480 14311 webserver.cc:460] Webserver started at http://127.0.98.2:46747/ using document root <none> and password file <none>
I20260924 05:11:20.466658 14311 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:20.466703 14311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:20.466837 14311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:20.467729 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "97e6fca1d7d34656aed551eb9bbccfeb"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "8b8e021d792c9492d21e6a69c0f3f744"
server_key_iv: "dae8a266aa57855ee60e9b0115b2b48a"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.467994 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "97e6fca1d7d34656aed551eb9bbccfeb"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "8b8e021d792c9492d21e6a69c0f3f744"
server_key_iv: "dae8a266aa57855ee60e9b0115b2b48a"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.469120 14311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:11:20.469767 14326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.469907 14311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:20.469980 14311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "97e6fca1d7d34656aed551eb9bbccfeb"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "8b8e021d792c9492d21e6a69c0f3f744"
server_key_iv: "dae8a266aa57855ee60e9b0115b2b48a"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.470027 14311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:20.480731 14311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:20.482383 14311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:20.482483 14311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:20.482632 14311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:20.482929 14311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:20.482962 14311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.482982 14311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:20.482995 14311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.488574 14311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34865
I20260924 05:11:20.488643 14439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34865 every 8 connection(s)
I20260924 05:11:20.488922 14311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:11:20.493633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 14311
I20260924 05:11:20.493750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:11:20.493932 392 external_mini_cluster.cc:1428] Setting key a1a428375306beb8f8344043ead9dd6e
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:20.498241 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.489638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:44703)
0924 05:11:20.489801 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:20.489836 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:20.490318 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:20.490571 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:20.490574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:20.491181 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:20.491363 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.491368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.492042 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.492043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.492793 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.492796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.493187 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.493188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.493306 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.493809 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:20.493836 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:20.495443 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:20.496941 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:20.496947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:20.496948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:20.497164 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:20.497394 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:20.497395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:20.497400 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:20.497500 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:20.497805 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:20.497809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:20.497909 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 05:11:20.497986 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 05:11:20.498091 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260924 05:11:20.498247 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.489622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44703 (local address 127.0.98.62:36995)
0924 05:11:20.489676 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.489678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.489922 (+ 244us) server_negotiation.cc:407] Connection header received
0924 05:11:20.490428 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.490429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.490441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.490505 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:20.491455 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.491951 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.492891 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.493094 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.493177 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.495531 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:20.495533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:20.495533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:20.495541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:20.496848 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.497242 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.497243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.497244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.497303 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.497573 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.497574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.497574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.497717 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:20.497788 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.498029 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.498116 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:11:20.498854 14440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36995
I20260924 05:11:20.498934 14440 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:20.499100 14440 heartbeater.cc:511] Master 127.0.98.62:36995 requested a full tablet report, sending...
I20260924 05:11:20.499467 14115 ts_manager.cc:194] Registered new tserver with Master: 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865)
I20260924 05:11:20.500003 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44703
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:11:20.517899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36995
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:20.595804 14445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:20.595980 14445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:20.596002 14445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:20.597131 14445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:20.597172 14445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:20.597224 14445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:11:20.598631 14445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:20.598876 14445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:20.599088 14445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:20.603657 14451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:20.603734 14453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:20.603772 14450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:20.603807 14445 server_base.cc:1034] running on GCE node
I20260924 05:11:20.604074 14445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:20.604276 14445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:20.605432 14445 hybrid_clock.cc:648] HybridClock initialized: now 1790226680605425 us; error 39 us; skew 500 ppm
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:20.607844 14445 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:11:20.608228 14445 webserver.cc:460] Webserver started at http://127.0.98.3:36369/ using document root <none> and password file <none>
I20260924 05:11:20.608393 14445 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:20.608430 14445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:20.608507 14445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:20.609418 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "5c7f7d45eec841aab64f64ecffe2c80e"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "d37c44ff938b5464bf551f3a64e59128"
server_key_iv: "3bda014803b09eb8e55e2d46bb05b77e"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.609697 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5c7f7d45eec841aab64f64ecffe2c80e"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "d37c44ff938b5464bf551f3a64e59128"
server_key_iv: "3bda014803b09eb8e55e2d46bb05b77e"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.610888 14445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:11:20.611519 14460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.611730 14445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.611832 14445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "5c7f7d45eec841aab64f64ecffe2c80e"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "d37c44ff938b5464bf551f3a64e59128"
server_key_iv: "3bda014803b09eb8e55e2d46bb05b77e"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.611905 14445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:20.622404 14445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:20.624502 14445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:20.624603 14445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:20.624812 14445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:20.625088 14445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:20.625120 14445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.625141 14445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:20.625186 14445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.631196 14445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36211
I20260924 05:11:20.631261 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36211 every 8 connection(s)
I20260924 05:11:20.631568 14445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:11:20.631904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 14445
I20260924 05:11:20.632014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:11:20.632232 392 external_mini_cluster.cc:1428] Setting key f9566ed5b9a17e4e957f35104ecfbb02
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:20.640538 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.632350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59959 (local address 127.0.98.62:36995)
0924 05:11:20.632401 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.632403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.632675 (+ 272us) server_negotiation.cc:407] Connection header received
0924 05:11:20.633317 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.633318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.633336 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.633380 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:20.634273 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.634805 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.635770 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.635983 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.636068 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.638014 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:20.638015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:20.638015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:20.638023 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:20.639217 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.639627 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.639628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.639629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.639684 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.639935 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.639936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.639936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.640044 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:20.640119 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.640378 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.640451 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:11:20.640717 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.632275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:59959)
0924 05:11:20.632544 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:20.632606 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:20.633232 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:20.633457 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:20.633461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:20.633995 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:20.634169 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.634172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.634877 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.634878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.635664 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.635666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.636076 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.636077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.636204 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.636631 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:20.636639 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:20.637923 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:20.639323 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:20.639329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:20.639332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:20.639538 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:20.639751 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:20.639752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:20.639753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:20.639857 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:20.640110 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:20.640116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:20.640248 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 05:11:20.640353 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 05:11:20.640500 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260924 05:11:20.641181 14574 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36995
I20260924 05:11:20.641271 14574 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:20.641435 14574 heartbeater.cc:511] Master 127.0.98.62:36995 requested a full tablet report, sending...
I20260924 05:11:20.641811 14115 ts_manager.cc:194] Registered new tserver with Master: 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211)
I20260924 05:11:20.642162 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59959
I20260924 05:11:20.643982 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:11:20.650496 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.645251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.0.98.62:36995)
0924 05:11:20.645288 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.645290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.645377 (+ 87us) server_negotiation.cc:407] Connection header received
0924 05:11:20.645418 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.645419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.645426 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.645475 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:20.645958 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.646393 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.647129 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.647307 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.647359 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.648344 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:20.648345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:20.648345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:20.648351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:20.649671 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.649936 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.649937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.649937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.649982 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:20.650114 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:20.650115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:20.650115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:20.650237 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:20.650300 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.650392 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.650441 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:11:20.651871 14115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:11:20.653504 14115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:11:20.659452 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.655499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40981 (local address 127.0.0.1:57756)
0924 05:11:20.655883 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:20.655894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:20.655947 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:20.656205 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:20.656206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:20.656208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:20.656251 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.656254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.657490 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.657491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.658261 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.658263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.659247 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.659247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.659301 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.659312 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 05:11:20.659368 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:11:20.659403 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260924 05:11:20.659502 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.655736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.0.98.1:40981)
0924 05:11:20.655917 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.655922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.655935 (+ 13us) server_negotiation.cc:407] Connection header received
0924 05:11:20.656032 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.656035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.656050 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.656117 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:20.656330 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.657413 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.658353 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.659163 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.659247 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.659299 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.659390 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.659424 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260924 05:11:20.660140 14240 tablet_service.cc:1467] Processing CreateTablet for tablet 34639c5bb504483cbd71f37afc804daf (DEFAULT_TABLE table=test-table [id=d21ed9cc5498487a95675ebf28b4faca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:20.660495 14240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34639c5bb504483cbd71f37afc804daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:20.660523 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.655499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36211 (local address 127.0.0.1:34690)
0924 05:11:20.655859 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:20.655867 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:20.655905 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:20.656073 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:20.656076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:20.656078 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:20.656144 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.656146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.657409 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.657409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.658822 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.658823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.660281 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.660282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.660350 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.660354 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:20.660412 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:11:20.660453 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":44,"thread_start_us":97,"threads_started":1}
I20260924 05:11:20.660722 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.655708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.0.98.3:36211)
0924 05:11:20.655884 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.655888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.655898 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:11:20.655974 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.655976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.655992 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.656065 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:20.656202 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.657319 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.659703 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.660204 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.660262 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.660291 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.660611 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.660655 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260924 05:11:20.663041 14588 tablet_bootstrap.cc:492] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Bootstrap starting.
I20260924 05:11:20.663493 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.655473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34865 (local address 127.0.0.1:56406)
0924 05:11:20.655645 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:20.655680 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:20.655743 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:20.656230 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:20.656230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:20.656232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:20.656275 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.656276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.657332 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.657333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.658820 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.658823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.663261 (+ 4438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.663262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.663325 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.663329 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:20.663389 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 05:11:20.663428 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":31,"thread_start_us":49,"threads_started":1}
I20260924 05:11:20.663573 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.655558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.0.98.2:34865)
0924 05:11:20.656015 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.656018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.656026 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:11:20.656063 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.656065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.656080 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.656153 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:20.656363 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.657214 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.658890 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.659540 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.659608 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.659667 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.663423 (+ 3756us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.663474 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":89,"threads_started":1}
I20260924 05:11:20.663623 14508 tablet_service.cc:1467] Processing CreateTablet for tablet 34639c5bb504483cbd71f37afc804daf (DEFAULT_TABLE table=test-table [id=d21ed9cc5498487a95675ebf28b4faca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:20.663981 14508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34639c5bb504483cbd71f37afc804daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:20.664184 14588 tablet_bootstrap.cc:654] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:20.664449 14374 tablet_service.cc:1467] Processing CreateTablet for tablet 34639c5bb504483cbd71f37afc804daf (DEFAULT_TABLE table=test-table [id=d21ed9cc5498487a95675ebf28b4faca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:20.664764 14374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34639c5bb504483cbd71f37afc804daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:20.664762 14588 log.cc:824] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:20.665562 14588 tablet_bootstrap.cc:492] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: No bootstrap required, opened a new log
I20260924 05:11:20.665624 14588 ts_tablet_manager.cc:1397] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 05:11:20.666320 14590 tablet_bootstrap.cc:492] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: Bootstrap starting.
I20260924 05:11:20.667025 14590 tablet_bootstrap.cc:654] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:20.667481 14590 log.cc:824] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:20.667989 14588 raft_consensus.cc:348] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:20.668159 14588 raft_consensus.cc:374] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:20.668185 14588 raft_consensus.cc:729] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35da93437b443ee943b114b6e2245a5, State: Initialized, Role: FOLLOWER
I20260924 05:11:20.668291 14590 tablet_bootstrap.cc:492] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: No bootstrap required, opened a new log
I20260924 05:11:20.668293 14588 consensus_queue.cc:260] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:20.668357 14590 ts_tablet_manager.cc:1397] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:11:20.668576 14306 heartbeater.cc:503] Master 127.0.98.62:36995 was elected leader, sending a full tablet report...
I20260924 05:11:20.668823 14588 ts_tablet_manager.cc:1428] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:11:20.669943 14590 raft_consensus.cc:348] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:20.670068 14590 raft_consensus.cc:374] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:20.670089 14590 raft_consensus.cc:729] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7f7d45eec841aab64f64ecffe2c80e, State: Initialized, Role: FOLLOWER
I20260924 05:11:20.670217 14590 consensus_queue.cc:260] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:20.670490 14590 ts_tablet_manager.cc:1428] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:11:20.670526 14574 heartbeater.cc:503] Master 127.0.98.62:36995 was elected leader, sending a full tablet report...
I20260924 05:11:20.670635 14592 tablet_bootstrap.cc:492] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: Bootstrap starting.
I20260924 05:11:20.671347 14592 tablet_bootstrap.cc:654] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:20.671789 14592 log.cc:824] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:20.672452 14592 tablet_bootstrap.cc:492] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: No bootstrap required, opened a new log
I20260924 05:11:20.672529 14592 ts_tablet_manager.cc:1397] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:11:20.673851 14592 raft_consensus.cc:348] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:20.674002 14592 raft_consensus.cc:374] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:20.674046 14592 raft_consensus.cc:729] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e6fca1d7d34656aed551eb9bbccfeb, State: Initialized, Role: FOLLOWER
I20260924 05:11:20.674154 14592 consensus_queue.cc:260] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:20.674388 14592 ts_tablet_manager.cc:1428] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:11:20.674417 14440 heartbeater.cc:503] Master 127.0.98.62:36995 was elected leader, sending a full tablet report...
W20260924 05:11:20.739457 14441 tablet.cc:2369] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:11:20.760787 14593 raft_consensus.cc:482] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:20.760901 14593 raft_consensus.cc:504] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:20.761303 14593 leader_election.cc:288] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865), 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211)
I20260924 05:11:20.765195 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.761563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60629 (local address 127.0.98.3:36211)
0924 05:11:20.761615 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.761618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.761783 (+ 165us) server_negotiation.cc:407] Connection header received
0924 05:11:20.761810 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.761811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.761823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.761873 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:20.762017 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.762755 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.764242 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.764834 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.764909 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.764918 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.765089 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.765126 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:11:20.765195 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.761655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52919 (local address 127.0.98.2:34865)
0924 05:11:20.761705 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.761707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.761713 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:11:20.761756 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.761757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.761769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.761839 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:20.762017 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.762753 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.764242 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.764841 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.764909 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.764918 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.765089 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.765126 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:11:20.765232 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.761467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34865 (local address 127.0.98.1:52919)
0924 05:11:20.761508 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:20.761520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:20.761578 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:20.761840 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:20.761841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:20.761842 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:20.761933 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.761935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.762857 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.762857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.764145 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.764148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.764922 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.764923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.765006 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.765008 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:11:20.765066 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:11:20.765121 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":34}
I20260924 05:11:20.765244 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.761467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36211 (local address 127.0.98.1:60629)
0924 05:11:20.761673 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:20.761684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:20.761748 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:20.761886 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:20.761887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:20.761888 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:20.761943 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.761944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.762857 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.762857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.764145 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:20.764148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.764922 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.764923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:20.765002 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.765007 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:11:20.765066 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:11:20.765121 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":118,"thread_start_us":92,"threads_started":1}
I20260924 05:11:20.765589 14394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34639c5bb504483cbd71f37afc804daf" candidate_uuid: "e35da93437b443ee943b114b6e2245a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" is_pre_election: true
I20260924 05:11:20.765728 14394 raft_consensus.cc:2454] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35da93437b443ee943b114b6e2245a5 in term 0.
I20260924 05:11:20.765856 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34639c5bb504483cbd71f37afc804daf" candidate_uuid: "e35da93437b443ee943b114b6e2245a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" is_pre_election: true
I20260924 05:11:20.766014 14528 raft_consensus.cc:2454] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35da93437b443ee943b114b6e2245a5 in term 0.
I20260924 05:11:20.766013 14195 leader_election.cc:302] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97e6fca1d7d34656aed551eb9bbccfeb, e35da93437b443ee943b114b6e2245a5; no voters:
I20260924 05:11:20.766135 14593 raft_consensus.cc:2781] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:11:20.766206 14593 raft_consensus.cc:482] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:11:20.766232 14593 raft_consensus.cc:3037] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:20.766963 14593 raft_consensus.cc:504] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:20.767100 14593 leader_election.cc:288] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 1 election: Requested vote from peers 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865), 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211)
I20260924 05:11:20.767262 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34639c5bb504483cbd71f37afc804daf" candidate_uuid: "e35da93437b443ee943b114b6e2245a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7f7d45eec841aab64f64ecffe2c80e"
I20260924 05:11:20.767321 14394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34639c5bb504483cbd71f37afc804daf" candidate_uuid: "e35da93437b443ee943b114b6e2245a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e6fca1d7d34656aed551eb9bbccfeb"
I20260924 05:11:20.767366 14528 raft_consensus.cc:3037] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:20.767380 14394 raft_consensus.cc:3037] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:20.768244 14528 raft_consensus.cc:2454] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35da93437b443ee943b114b6e2245a5 in term 1.
I20260924 05:11:20.768244 14394 raft_consensus.cc:2454] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35da93437b443ee943b114b6e2245a5 in term 1.
I20260924 05:11:20.768429 14196 leader_election.cc:302] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c7f7d45eec841aab64f64ecffe2c80e, e35da93437b443ee943b114b6e2245a5; no voters:
I20260924 05:11:20.768515 14593 raft_consensus.cc:2781] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:20.768656 14593 raft_consensus.cc:686] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 1 LEADER]: Becoming Leader. State: Replica: e35da93437b443ee943b114b6e2245a5, State: Running, Role: LEADER
I20260924 05:11:20.768760 14593 consensus_queue.cc:237] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:20.769511 14115 catalog_manager.cc:5495] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e35da93437b443ee943b114b6e2245a5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e35da93437b443ee943b114b6e2245a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } health_report { overall_health: HEALTHY } } }
I20260924 05:11:20.802814 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.800489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.98.1:40981)
0924 05:11:20.800535 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:20.800537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:20.800546 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:11:20.800628 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:20.800631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:20.800654 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:20.800740 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:11:20.800881 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.801329 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.802040 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:20.802222 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:20.802291 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:20.802469 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:11:20.802560 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:11:20.802649 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:20.802672 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 05:11:20.802692 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:11:20.804539 14528 raft_consensus.cc:1261] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [term 1 FOLLOWER]: Refusing update from remote peer e35da93437b443ee943b114b6e2245a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:20.804564 14394 raft_consensus.cc:1261] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [term 1 FOLLOWER]: Refusing update from remote peer e35da93437b443ee943b114b6e2245a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:20.804929 14593 consensus_queue.cc:1035] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:20.805073 14598 consensus_queue.cc:1035] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:20.807598 14601 mvcc.cc:205] Tried to move back new op lower bound from 7332768484572495872 to 7332768484429484032. Current Snapshot: MvccSnapshot[applied={T|T < 7332768484572495872}]
I20260924 05:11:20.809628 14114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:20.809718 14114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:20.810482 14114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Sending DeleteTablet for 3 replicas of tablet 34639c5bb504483cbd71f37afc804daf
I20260924 05:11:20.810819 14374 tablet_service.cc:1514] Processing DeleteTablet for tablet 34639c5bb504483cbd71f37afc804daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56406
I20260924 05:11:20.810840 14508 tablet_service.cc:1514] Processing DeleteTablet for tablet 34639c5bb504483cbd71f37afc804daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34690
I20260924 05:11:20.810932 14240 tablet_service.cc:1514] Processing DeleteTablet for tablet 34639c5bb504483cbd71f37afc804daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57756
I20260924 05:11:20.811076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 14177
I20260924 05:11:20.811774 14611 tablet_replica.cc:330] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: stopping tablet replica
I20260924 05:11:20.811844 14610 tablet_replica.cc:330] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: stopping tablet replica
I20260924 05:11:20.811913 14611 raft_consensus.cc:2229] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:20.811993 14610 raft_consensus.cc:2229] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:20.812041 14611 raft_consensus.cc:2258] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:20.812132 14610 raft_consensus.cc:2258] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:20.812505 14611 ts_tablet_manager.cc:1905] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:20.812543 14610 ts_tablet_manager.cc:1905] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:20.814183 14611 ts_tablet_manager.cc:1918] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:20.814247 14610 ts_tablet_manager.cc:1918] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:20.814293 14611 log.cc:1196] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/34639c5bb504483cbd71f37afc804daf
I20260924 05:11:20.814323 14610 log.cc:1196] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/34639c5bb504483cbd71f37afc804daf
I20260924 05:11:20.814607 14610 ts_tablet_manager.cc:1939] T 34639c5bb504483cbd71f37afc804daf P 97e6fca1d7d34656aed551eb9bbccfeb: Deleting consensus metadata
I20260924 05:11:20.814607 14611 ts_tablet_manager.cc:1939] T 34639c5bb504483cbd71f37afc804daf P 5c7f7d45eec841aab64f64ecffe2c80e: Deleting consensus metadata
I20260924 05:11:20.815001 14101 catalog_manager.cc:4841] TS 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865): tablet 34639c5bb504483cbd71f37afc804daf (table test-table [id=d21ed9cc5498487a95675ebf28b4faca]) successfully deleted
I20260924 05:11:20.815011 14102 catalog_manager.cc:4841] TS 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211): tablet 34639c5bb504483cbd71f37afc804daf (table test-table [id=d21ed9cc5498487a95675ebf28b4faca]) successfully deleted
W20260924 05:11:20.816821 14099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40981 (error 108)
I20260924 05:11:20.817041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 14311
I20260924 05:11:20.817227 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:20.817094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40981 (local address 127.0.0.1:57776)
0924 05:11:20.817148 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:20.817193 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 05:11:20.817929 14099 catalog_manager.cc:4584] TS e35da93437b443ee943b114b6e2245a5 (127.0.98.1:40981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34639c5bb504483cbd71f37afc804daf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40981: connect: Connection refused (error 111)
I20260924 05:11:20.821709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 14445
I20260924 05:11:20.826119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 14083
I20260924 05:11:20.831012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36995
--webserver_interface=127.0.98.62
--webserver_port=40727
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:20.907749 14614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:20.907933 14614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:20.907955 14614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:20.909111 14614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:20.909152 14614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:20.909165 14614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:20.909178 14614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:20.909189 14614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:20.910578 14614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:20.910812 14614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:20.911018 14614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:20.919256 14623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:20.919272 14620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:20.919265 14621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:20.919298 14614 server_base.cc:1034] running on GCE node
I20260924 05:11:20.919744 14614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:20.920023 14614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:20.921175 14614 hybrid_clock.cc:648] HybridClock initialized: now 1790226680921165 us; error 36 us; skew 500 ppm
Sep 24 05:11:20 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:20.923584 14614 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:11:20.924000 14614 webserver.cc:460] Webserver started at http://127.0.98.62:40727/ using document root <none> and password file <none>
I20260924 05:11:20.924227 14614 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:20.924300 14614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:20.925637 14614 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:11:20.926220 14630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:20.926388 14614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:11:20.926513 14614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "327fd3ed012041d8830a50b2e21bc180"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "f0518dc4da732db4a87bf42489b84bee"
server_key_iv: "7c24df4da608a58c7f20a5c09c32c0f3"
server_key_version: "encryptionkey@0"
I20260924 05:11:20.926930 14614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:20.933912 14614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:20.935683 14614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:20.935828 14614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:20.939620 14614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36995
I20260924 05:11:20.939648 14682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36995 every 8 connection(s)
I20260924 05:11:20.940054 14614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:11:20.941526 14683 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:11:20.942157 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Bootstrap starting.
I20260924 05:11:20.945358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 14614
I20260924 05:11:20.945696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40981
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36995
--webserver_port=43913
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:20.951045 14683 log.cc:824] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:20.952884 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:11:20.953161 14683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Bootstrap complete.
I20260924 05:11:20.955464 14683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:20.955711 14683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327fd3ed012041d8830a50b2e21bc180, State: Initialized, Role: FOLLOWER
I20260924 05:11:20.955909 14683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:20.956005 14683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:11:20.956061 14683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:11:20.956110 14683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:11:20.957077 14683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:20.957193 14683 leader_election.cc:302] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 327fd3ed012041d8830a50b2e21bc180; no voters:
I20260924 05:11:20.957383 14683 leader_election.cc:288] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:11:20.957414 14687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:11:20.957527 14687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 2 LEADER]: Becoming Leader. State: Replica: 327fd3ed012041d8830a50b2e21bc180, State: Running, Role: LEADER
I20260924 05:11:20.957638 14683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:20.957674 14687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:20.958019 14689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 327fd3ed012041d8830a50b2e21bc180. Latest consensus state: current_term: 2 leader_uuid: "327fd3ed012041d8830a50b2e21bc180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } } }
I20260924 05:11:20.958029 14688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "327fd3ed012041d8830a50b2e21bc180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } } }
I20260924 05:11:20.958109 14689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:20.958140 14688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:20.958943 14694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:11:20.959910 14694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:11:20.960001 14694 catalog_manager.cc:1259] Loaded cluster ID: 67437826f29449948c96324a318f43c3
I20260924 05:11:20.960031 14694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:11:20.960287 14694 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 05:11:20.965286 14694 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 05:11:20.967635 14694 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:11:20.967821 14694 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Loaded TSK: 0
I20260924 05:11:20.968029 14694 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:11:21.028539 14685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:21.028728 14685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:21.028749 14685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:21.029978 14685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:21.030038 14685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:21.030068 14685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:11:21.031570 14685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43913
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:21.031798 14685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:21.032068 14685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:21.036819 14713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:21.036847 14710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:21.036808 14711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:21.037216 14685 server_base.cc:1034] running on GCE node
I20260924 05:11:21.037439 14685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:21.037689 14685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:21.038817 14685 hybrid_clock.cc:648] HybridClock initialized: now 1790226681038803 us; error 30 us; skew 500 ppm
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:21.041239 14685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:11:21.041688 14685 webserver.cc:460] Webserver started at http://127.0.98.1:43913/ using document root <none> and password file <none>
I20260924 05:11:21.041901 14685 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:21.041951 14685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:21.043256 14685 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:11:21.043996 14720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.044176 14685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:11:21.044282 14685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "e35da93437b443ee943b114b6e2245a5"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "98651adbb1d6e5013246c878b526f084"
server_key_iv: "3bfbe7103129eacc7733cdc8a0ac2225"
server_key_version: "encryptionkey@0"
I20260924 05:11:21.044616 14685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:21.059389 14685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:21.061463 14685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:21.061616 14685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:21.061831 14685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:21.062254 14727 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:11:21.064100 14685 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:11:21.064159 14685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:11:21.064204 14685 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:11:21.064805 14685 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:11:21.064862 14685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:11:21.064919 14727 tablet_bootstrap.cc:492] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Bootstrap starting.
I20260924 05:11:21.071908 14685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40981
I20260924 05:11:21.071947 14834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40981 every 8 connection(s)
I20260924 05:11:21.072309 14685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:11:21.075044 14727 log.cc:824] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:21.077719 14727 tablet_bootstrap.cc:492] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:11:21.078017 14727 tablet_bootstrap.cc:492] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Bootstrap complete.
I20260924 05:11:21.079246 14727 ts_tablet_manager.cc:1397] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260924 05:11:21.080725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 14685
I20260924 05:11:21.080627 14727 raft_consensus.cc:348] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.080994 14727 raft_consensus.cc:729] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35da93437b443ee943b114b6e2245a5, State: Initialized, Role: FOLLOWER
I20260924 05:11:21.081113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34865
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36995
--webserver_port=46747
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:21.081226 14727 consensus_queue.cc:260] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.081581 14727 ts_tablet_manager.cc:1428] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:21.091570 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.072905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:58477)
0924 05:11:21.074889 (+ 1984us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.074924 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:21.075552 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:21.076372 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:21.076376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:21.076676 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:21.076856 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.076860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.078063 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.078064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.078928 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.078930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.079560 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.079561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.079693 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.085620 (+ 5927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:21.085639 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:21.087579 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:21.090033 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:21.090038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:21.090040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:21.090251 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:21.090548 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:21.090548 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:21.090549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:21.090636 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:21.091127 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:21.091132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:21.091240 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 05:11:21.091314 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 05:11:21.091429 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1947,"thread_start_us":114,"threads_started":1}
I20260924 05:11:21.091825 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.073035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58477 (local address 127.0.98.62:36995)
0924 05:11:21.076013 (+ 2978us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.076021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.076048 (+ 27us) server_negotiation.cc:407] Connection header received
0924 05:11:21.076148 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.076152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.076191 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.076338 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:21.076980 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.077954 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.079020 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.079446 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.079556 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.087713 (+ 8157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:21.087717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:21.087718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:21.087732 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:21.089937 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.090356 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.090360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.090361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.090478 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.090711 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.090712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.090713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.091056 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:21.091151 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.091429 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.091577 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2935,"thread_start_us":65,"threads_started":1}
I20260924 05:11:21.092459 14835 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36995
I20260924 05:11:21.092563 14835 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:21.092751 14835 heartbeater.cc:511] Master 127.0.98.62:36995 requested a full tablet report, sending...
I20260924 05:11:21.093446 14647 ts_manager.cc:194] Registered new tserver with Master: e35da93437b443ee943b114b6e2245a5 (127.0.98.1:40981)
I20260924 05:11:21.094581 14647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58477
I20260924 05:11:21.095441 14835 heartbeater.cc:503] Master 127.0.98.62:36995 was elected leader, sending a full tablet report...
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:11:21.102023 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.094013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40981 (local address 127.0.0.1:57784)
0924 05:11:21.094279 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.094304 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:21.094390 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:21.094583 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:21.094588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:21.094958 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:21.095017 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.095019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.096278 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.096279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.096837 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.096839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.097442 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.097445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.097492 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.097930 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:21.097935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:21.099226 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:21.101033 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:21.101038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:21.101039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:21.101220 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:21.101433 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:21.101434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:21.101434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:21.101483 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:21.101758 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:21.101763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:21.101801 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 05:11:21.101871 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:11:21.101949 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":91,"threads_started":1}
I20260924 05:11:21.102128 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.094064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.0.98.1:40981)
0924 05:11:21.094230 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.094233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.094372 (+ 139us) server_negotiation.cc:407] Connection header received
0924 05:11:21.094484 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.094486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.094500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.094579 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:21.095098 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.096205 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.096913 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.097371 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.097441 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.099301 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:21.099308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:21.099308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:21.099316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:21.100952 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.101297 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.101300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.101300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.101367 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.101540 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.101541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.101541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.101697 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:21.101775 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.101928 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.102030 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260924 05:11:21.102424 14769 tablet_service.cc:1514] Processing DeleteTablet for tablet 34639c5bb504483cbd71f37afc804daf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57784
I20260924 05:11:21.103093 14844 tablet_replica.cc:330] stopping tablet replica
I20260924 05:11:21.103204 14844 raft_consensus.cc:2229] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:21.103297 14844 raft_consensus.cc:2258] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:21.103567 14844 ts_tablet_manager.cc:1905] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:21.105532 14844 ts_tablet_manager.cc:1918] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:21.105589 14844 log.cc:1196] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/34639c5bb504483cbd71f37afc804daf
I20260924 05:11:21.105806 14844 ts_tablet_manager.cc:1939] T 34639c5bb504483cbd71f37afc804daf P e35da93437b443ee943b114b6e2245a5: Deleting consensus metadata
I20260924 05:11:21.106230 14631 catalog_manager.cc:4841] TS e35da93437b443ee943b114b6e2245a5 (127.0.98.1:40981): tablet 34639c5bb504483cbd71f37afc804daf (table test-table [id=d21ed9cc5498487a95675ebf28b4faca]) successfully deleted
W20260924 05:11:21.166118 14841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:21.166355 14841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:21.166395 14841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:21.167603 14841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:21.167671 14841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:21.167716 14841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:11:21.169147 14841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46747
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:21.169412 14841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:21.169642 14841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:21.174471 14852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:21.174458 14849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:21.174573 14841 server_base.cc:1034] running on GCE node
W20260924 05:11:21.174440 14850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:21.174888 14841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:21.175143 14841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:21.176301 14841 hybrid_clock.cc:648] HybridClock initialized: now 1790226681176298 us; error 36 us; skew 500 ppm
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:21.178594 14841 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:11:21.179069 14841 webserver.cc:460] Webserver started at http://127.0.98.2:46747/ using document root <none> and password file <none>
I20260924 05:11:21.179296 14841 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:21.179373 14841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:21.180673 14841 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:11:21.181234 14859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.181397 14841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:11:21.181504 14841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "97e6fca1d7d34656aed551eb9bbccfeb"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "8b8e021d792c9492d21e6a69c0f3f744"
server_key_iv: "dae8a266aa57855ee60e9b0115b2b48a"
server_key_version: "encryptionkey@0"
I20260924 05:11:21.181819 14841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:21.192243 14841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:21.193938 14841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:21.194100 14841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:21.194317 14841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:21.194607 14841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:21.194660 14841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.194694 14841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:21.194721 14841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.200398 14841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34865
I20260924 05:11:21.200441 14972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34865 every 8 connection(s)
I20260924 05:11:21.200753 14841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:11:21.206079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 14841
I20260924 05:11:21.206369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36211
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36995
--webserver_port=36369
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:21.209830 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.201463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48215 (local address 127.0.98.62:36995)
0924 05:11:21.201565 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.201568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.201741 (+ 173us) server_negotiation.cc:407] Connection header received
0924 05:11:21.202173 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.202174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.202186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.202236 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:21.203174 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.203657 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.204581 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.204768 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.204818 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.206999 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:21.207001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:21.207001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:21.207009 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:21.208411 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.208911 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.208912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.208912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.208958 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.209228 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.209228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.209228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.209318 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:21.209372 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.209698 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.209767 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 05:11:21.210112 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.201437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:48215)
0924 05:11:21.201626 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.201658 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:21.202096 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:21.202284 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:21.202287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:21.202893 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:21.203094 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.203099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.203734 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.203734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.204509 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.204511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.204836 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.204837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.204929 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.205451 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:21.205459 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:21.206909 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:21.208500 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:21.208508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:21.208511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:21.208826 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:21.209021 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:21.209022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:21.209022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:21.209158 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:21.209386 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:21.209391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:21.209568 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 05:11:21.209682 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 05:11:21.209858 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":122,"threads_started":1}
I20260924 05:11:21.210789 14973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36995
I20260924 05:11:21.210906 14973 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:21.211107 14973 heartbeater.cc:511] Master 127.0.98.62:36995 requested a full tablet report, sending...
I20260924 05:11:21.211474 14647 ts_manager.cc:194] Registered new tserver with Master: 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865)
I20260924 05:11:21.211863 14647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48215
W20260924 05:11:21.285934 14976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:21.286151 14976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:21.286188 14976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:21.287335 14976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:21.287401 14976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:21.287446 14976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:11:21.288851 14976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36369
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:21.289099 14976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:21.289317 14976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:21.294066 14984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:21.294075 14981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:21.294207 14982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:21.294221 14976 server_base.cc:1034] running on GCE node
I20260924 05:11:21.294493 14976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:21.294711 14976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:21.295893 14976 hybrid_clock.cc:648] HybridClock initialized: now 1790226681295894 us; error 41 us; skew 500 ppm
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:21.298292 14976 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:11:21.298647 14976 webserver.cc:460] Webserver started at http://127.0.98.3:36369/ using document root <none> and password file <none>
I20260924 05:11:21.298883 14976 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:21.298938 14976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:21.300127 14976 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:21.300690 14991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.300859 14976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:21.300945 14976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "5c7f7d45eec841aab64f64ecffe2c80e"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "d37c44ff938b5464bf551f3a64e59128"
server_key_iv: "3bda014803b09eb8e55e2d46bb05b77e"
server_key_version: "encryptionkey@0"
I20260924 05:11:21.301285 14976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:21.308013 14976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:21.310052 14976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:21.310151 14976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:21.310307 14976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:21.310601 14976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:21.310635 14976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.310655 14976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:21.310711 14976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.316222 14976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36211
I20260924 05:11:21.316273 15104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36211 every 8 connection(s)
I20260924 05:11:21.316619 14976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:11:21.321658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 14976
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:21.325539 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.317316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34249 (local address 127.0.98.62:36995)
0924 05:11:21.317394 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.317396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.317543 (+ 147us) server_negotiation.cc:407] Connection header received
0924 05:11:21.317987 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.317988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.317998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.318048 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:21.318909 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.319362 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.320213 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.320374 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.320419 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.322578 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:21.322578 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:21.322579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:21.322584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:21.324070 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.324531 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.324532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.324532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.324594 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.324857 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.324858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.324858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.324975 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:21.325046 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.325378 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.325454 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 05:11:21.325703 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.317277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:34249)
0924 05:11:21.317435 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.317467 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:21.317912 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:21.318095 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:21.318098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:21.318598 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:21.318830 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.318842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.319432 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.319433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.320150 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.320152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.320436 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.320437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.320521 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.320962 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:21.320969 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:21.322493 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:21.324141 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:21.324148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:21.324150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:21.324431 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:21.324671 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:21.324672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:21.324672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:21.324800 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:21.325061 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:21.325067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:21.325214 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 05:11:21.325350 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 05:11:21.325497 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":101,"threads_started":1}
I20260924 05:11:21.326134 15105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36995
I20260924 05:11:21.326239 15105 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:21.326418 15105 heartbeater.cc:511] Master 127.0.98.62:36995 requested a full tablet report, sending...
I20260924 05:11:21.326896 14647 ts_manager.cc:194] Registered new tserver with Master: 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211)
I20260924 05:11:21.327394 14647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34249
I20260924 05:11:21.328977 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.322057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.0.98.62:36995)
0924 05:11:21.322310 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.322312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.322317 (+ 5us) server_negotiation.cc:407] Connection header received
0924 05:11:21.322333 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.322334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.322353 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.322395 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:21.323050 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.323791 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.324746 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.324998 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.325056 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.326415 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:21.326416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:21.326417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:21.326422 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:21.327934 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.328297 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.328297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.328298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.328352 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:21.328493 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:21.328494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:21.328494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:21.328569 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:21.328721 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.328755 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.328868 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":65,"threads_started":1}
I20260924 05:11:21.329632 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:11:21.332187 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.330014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.0.98.62:36995)
0924 05:11:21.330047 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.330049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.330174 (+ 125us) server_negotiation.cc:407] Connection header received
0924 05:11:21.330226 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.330227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.330249 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.330305 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:11:21.330396 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.330912 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.331591 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.331757 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.331803 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.331927 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:11:21.332004 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:11:21.332059 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.332089 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.332104 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:11:21.332417 14647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:11:21.333842 14647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:11:21.335881 14769 tablet_service.cc:1467] Processing CreateTablet for tablet 214b8340f4a5491aa87bc3b28692d8b6 (DEFAULT_TABLE table=test-table [id=733caa5a60bc4f49b2b69cd652864de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:21.336074 14769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 214b8340f4a5491aa87bc3b28692d8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:21.337649 14727 tablet_bootstrap.cc:492] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Bootstrap starting.
I20260924 05:11:21.338086 14727 tablet_bootstrap.cc:654] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:21.338687 14727 tablet_bootstrap.cc:492] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: No bootstrap required, opened a new log
I20260924 05:11:21.338752 14727 ts_tablet_manager.cc:1397] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:11:21.339048 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.335518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36211 (local address 127.0.0.1:34698)
0924 05:11:21.335550 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.335559 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:21.335593 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:21.335911 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:21.335912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:21.335914 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:21.335959 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.335960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.336943 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.336944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.337925 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.337928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.338832 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.338833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.338901 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.338921 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 05:11:21.338950 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 05:11:21.338988 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 05:11:21.339088 14727 raft_consensus.cc:348] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.339174 14727 raft_consensus.cc:374] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:21.339219 14727 raft_consensus.cc:729] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35da93437b443ee943b114b6e2245a5, State: Initialized, Role: FOLLOWER
I20260924 05:11:21.339290 14727 consensus_queue.cc:260] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.339404 14727 ts_tablet_manager.cc:1428] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:11:21.339470 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.335589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.0.98.3:36211)
0924 05:11:21.335740 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.335744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.335752 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:11:21.335784 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.335786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.335801 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.335903 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:21.336012 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.336839 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.338027 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.338713 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.339288 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.339339 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.339365 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.339403 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260924 05:11:21.339541 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.335518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34865 (local address 127.0.0.1:56408)
0924 05:11:21.335856 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.335866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:21.335923 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:21.336124 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:21.336125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:21.336127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:21.336158 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.336159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.337369 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.337370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.338402 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.338404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.339294 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.339295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.339359 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.339363 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:21.339422 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:11:21.339460 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":21,"thread_start_us":80,"threads_started":1}
I20260924 05:11:21.339980 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.335688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.0.98.2:34865)
0924 05:11:21.335835 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.335839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.335979 (+ 140us) server_negotiation.cc:407] Connection header received
0924 05:11:21.336021 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.336024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.336039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.336116 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:21.336240 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.337260 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.338489 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.339209 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.339283 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.339339 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.339688 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.339731 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260924 05:11:21.340049 15039 tablet_service.cc:1467] Processing CreateTablet for tablet 214b8340f4a5491aa87bc3b28692d8b6 (DEFAULT_TABLE table=test-table [id=733caa5a60bc4f49b2b69cd652864de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:21.340402 15039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 214b8340f4a5491aa87bc3b28692d8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:21.340631 14907 tablet_service.cc:1467] Processing CreateTablet for tablet 214b8340f4a5491aa87bc3b28692d8b6 (DEFAULT_TABLE table=test-table [id=733caa5a60bc4f49b2b69cd652864de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:21.340927 14907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 214b8340f4a5491aa87bc3b28692d8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:21.343837 15115 tablet_bootstrap.cc:492] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: Bootstrap starting.
I20260924 05:11:21.344231 15116 tablet_bootstrap.cc:492] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Bootstrap starting.
I20260924 05:11:21.344461 15115 tablet_bootstrap.cc:654] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:21.344864 15116 tablet_bootstrap.cc:654] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:21.344982 15115 log.cc:824] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:21.345405 15116 log.cc:824] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:21.345775 15115 tablet_bootstrap.cc:492] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: No bootstrap required, opened a new log
I20260924 05:11:21.345866 15115 ts_tablet_manager.cc:1397] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:11:21.346094 15116 tablet_bootstrap.cc:492] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: No bootstrap required, opened a new log
I20260924 05:11:21.346144 15116 ts_tablet_manager.cc:1397] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:11:21.347419 15115 raft_consensus.cc:348] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.347570 15115 raft_consensus.cc:374] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:21.347615 15115 raft_consensus.cc:729] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7f7d45eec841aab64f64ecffe2c80e, State: Initialized, Role: FOLLOWER
I20260924 05:11:21.347786 15115 consensus_queue.cc:260] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.347832 15116 raft_consensus.cc:348] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.348047 15105 heartbeater.cc:503] Master 127.0.98.62:36995 was elected leader, sending a full tablet report...
I20260924 05:11:21.348053 15115 ts_tablet_manager.cc:1428] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:11:21.347993 15116 raft_consensus.cc:374] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:21.348232 15116 raft_consensus.cc:729] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e6fca1d7d34656aed551eb9bbccfeb, State: Initialized, Role: FOLLOWER
I20260924 05:11:21.348336 15116 consensus_queue.cc:260] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.348629 15116 ts_tablet_manager.cc:1428] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:11:21.348649 14973 heartbeater.cc:503] Master 127.0.98.62:36995 was elected leader, sending a full tablet report...
I20260924 05:11:21.433835 15120 raft_consensus.cc:482] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:21.433946 15120 raft_consensus.cc:504] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.434420 15120 leader_election.cc:288] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211), e35da93437b443ee943b114b6e2245a5 (127.0.98.1:40981)
I20260924 05:11:21.437188 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.434636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38401 (local address 127.0.98.3:36211)
0924 05:11:21.434700 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.434703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.434728 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:11:21.434777 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.434778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.434790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.434846 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:21.435032 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.435482 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.436354 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.436837 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.436928 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.436938 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.437089 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.437125 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:11:21.437197 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.434555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36211 (local address 127.0.98.2:38401)
0924 05:11:21.434623 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.434635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:21.434681 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:21.434893 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:21.434894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:21.434895 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:21.434968 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.434970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.435566 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.435567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.436262 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.436264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.436928 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.436929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.437011 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.437014 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:11:21.437082 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:11:21.437121 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 05:11:21.437371 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.434761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52317 (local address 127.0.98.1:40981)
0924 05:11:21.434837 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.434839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.434848 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:11:21.434874 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.434875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.434900 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.434969 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:21.435164 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.435648 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.436565 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.436975 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.437059 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.437135 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.437223 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.437266 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 05:11:21.437512 15059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "214b8340f4a5491aa87bc3b28692d8b6" candidate_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" is_pre_election: true
I20260924 05:11:21.437641 15059 raft_consensus.cc:2454] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e6fca1d7d34656aed551eb9bbccfeb in term 0.
I20260924 05:11:21.437773 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.434590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40981 (local address 127.0.98.2:52317)
0924 05:11:21.434777 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.434785 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:21.434823 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:21.435006 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:21.435007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:21.435009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:21.435088 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.435090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.435745 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.435745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.436457 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:21.436459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.437076 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.437077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:21.437155 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.437159 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:21.437216 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 05:11:21.437721 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":120,"threads_started":1}
I20260924 05:11:21.437839 14863 leader_election.cc:302] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c7f7d45eec841aab64f64ecffe2c80e, 97e6fca1d7d34656aed551eb9bbccfeb; no voters:
I20260924 05:11:21.437956 15120 raft_consensus.cc:2781] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:11:21.438016 15120 raft_consensus.cc:482] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:11:21.438042 15120 raft_consensus.cc:3037] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:21.438083 14789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "214b8340f4a5491aa87bc3b28692d8b6" candidate_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35da93437b443ee943b114b6e2245a5" is_pre_election: true
I20260924 05:11:21.438225 14789 raft_consensus.cc:2454] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e6fca1d7d34656aed551eb9bbccfeb in term 0.
I20260924 05:11:21.438886 15120 raft_consensus.cc:504] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.439011 15120 leader_election.cc:288] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [CANDIDATE]: Term 1 election: Requested vote from peers 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211), e35da93437b443ee943b114b6e2245a5 (127.0.98.1:40981)
I20260924 05:11:21.439162 15059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "214b8340f4a5491aa87bc3b28692d8b6" candidate_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7f7d45eec841aab64f64ecffe2c80e"
I20260924 05:11:21.439167 14789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "214b8340f4a5491aa87bc3b28692d8b6" candidate_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35da93437b443ee943b114b6e2245a5"
I20260924 05:11:21.439230 14789 raft_consensus.cc:3037] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:21.439249 15059 raft_consensus.cc:3037] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:21.439883 14789 raft_consensus.cc:2454] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e6fca1d7d34656aed551eb9bbccfeb in term 1.
I20260924 05:11:21.440047 14860 leader_election.cc:302] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97e6fca1d7d34656aed551eb9bbccfeb, e35da93437b443ee943b114b6e2245a5; no voters:
I20260924 05:11:21.440047 15059 raft_consensus.cc:2454] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e6fca1d7d34656aed551eb9bbccfeb in term 1.
I20260924 05:11:21.440143 15120 raft_consensus.cc:2781] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:21.440295 15120 raft_consensus.cc:686] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 1 LEADER]: Becoming Leader. State: Replica: 97e6fca1d7d34656aed551eb9bbccfeb, State: Running, Role: LEADER
I20260924 05:11:21.440385 15120 consensus_queue.cc:237] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.441138 14646 catalog_manager.cc:5495] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } health_report { overall_health: UNKNOWN } } }
W20260924 05:11:21.451261 14974 tablet.cc:2369] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:11:21.458235 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.455764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.0.98.2:34865)
0924 05:11:21.455818 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:21.455820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:21.455831 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:11:21.455863 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:21.455864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:21.455882 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:21.455949 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:11:21.456065 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.456503 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.457341 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:21.457544 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:21.457618 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:21.457762 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:11:21.457923 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:11:21.457999 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:21.458065 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 05:11:21.458089 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:11:21.459947 14789 raft_consensus.cc:1261] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Refusing update from remote peer 97e6fca1d7d34656aed551eb9bbccfeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:21.459949 15059 raft_consensus.cc:1261] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [term 1 FOLLOWER]: Refusing update from remote peer 97e6fca1d7d34656aed551eb9bbccfeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:21.460264 15122 consensus_queue.cc:1035] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:21.460386 15120 consensus_queue.cc:1035] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:21.462038 15125 mvcc.cc:205] Tried to move back new op lower bound from 7332768487257677824 to 7332768487180464128. Current Snapshot: MvccSnapshot[applied={T|T < 7332768487257677824}]
I20260924 05:11:21.462740 15128 mvcc.cc:205] Tried to move back new op lower bound from 7332768487257677824 to 7332768487180464128. Current Snapshot: MvccSnapshot[applied={T|T < 7332768487257677824}]
I20260924 05:11:21.463032 15126 mvcc.cc:205] Tried to move back new op lower bound from 7332768487257677824 to 7332768487180464128. Current Snapshot: MvccSnapshot[applied={T|T < 7332768487257677824}]
I20260924 05:11:21.464957 14647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:21.465031 14647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:21.465638 14647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Sending DeleteTablet for 3 replicas of tablet 214b8340f4a5491aa87bc3b28692d8b6
I20260924 05:11:21.465890 14907 tablet_service.cc:1514] Processing DeleteTablet for tablet 214b8340f4a5491aa87bc3b28692d8b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56408
I20260924 05:11:21.465932 14769 tablet_service.cc:1514] Processing DeleteTablet for tablet 214b8340f4a5491aa87bc3b28692d8b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57784
I20260924 05:11:21.465942 15039 tablet_service.cc:1514] Processing DeleteTablet for tablet 214b8340f4a5491aa87bc3b28692d8b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34698
I20260924 05:11:21.466015 14844 tablet_replica.cc:330] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: stopping tablet replica
I20260924 05:11:21.466080 14844 raft_consensus.cc:2229] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:21.466158 14844 raft_consensus.cc:2258] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:21.466213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 14685
I20260924 05:11:21.466468 14844 ts_tablet_manager.cc:1905] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:21.470880 15135 tablet_replica.cc:330] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: stopping tablet replica
I20260924 05:11:21.470981 15135 raft_consensus.cc:2229] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:21.471078 15135 raft_consensus.cc:2258] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:21.471098 15134 tablet_replica.cc:330] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: stopping tablet replica
I20260924 05:11:21.471263 15134 raft_consensus.cc:2229] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:11:21.471402 15134 raft_consensus.cc:2258] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:21.471493 15135 ts_tablet_manager.cc:1905] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:21.471843 15134 ts_tablet_manager.cc:1905] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:11:21.472066 14631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40981 (error 108)
I20260924 05:11:21.472888 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.472785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40981 (local address 127.0.0.1:57786)
0924 05:11:21.472828 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.472855 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:11:21.473171 15135 ts_tablet_manager.cc:1918] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:21.473250 15135 log.cc:1196] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/214b8340f4a5491aa87bc3b28692d8b6
W20260924 05:11:21.472999 14631 catalog_manager.cc:4584] TS e35da93437b443ee943b114b6e2245a5 (127.0.98.1:40981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 214b8340f4a5491aa87bc3b28692d8b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40981: connect: Connection refused (error 111)
I20260924 05:11:21.473534 15135 ts_tablet_manager.cc:1939] T 214b8340f4a5491aa87bc3b28692d8b6 P 5c7f7d45eec841aab64f64ecffe2c80e: Deleting consensus metadata
I20260924 05:11:21.473651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 14841
I20260924 05:11:21.473770 15134 ts_tablet_manager.cc:1918] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:21.473860 15134 log.cc:1196] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/214b8340f4a5491aa87bc3b28692d8b6
I20260924 05:11:21.474112 15134 ts_tablet_manager.cc:1939] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Deleting consensus metadata
I20260924 05:11:21.474375 14634 catalog_manager.cc:4841] TS 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211): tablet 214b8340f4a5491aa87bc3b28692d8b6 (table test-table [id=733caa5a60bc4f49b2b69cd652864de7]) successfully deleted
I20260924 05:11:21.477975 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.477864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34865 (local address 127.0.0.1:56436)
0924 05:11:21.477915 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.477942 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:11:21.478163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 14976
I20260924 05:11:21.483062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 14614
I20260924 05:11:21.487293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36995
--webserver_interface=127.0.98.62
--webserver_port=40727
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:21.565681 15137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:21.565871 15137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:21.565891 15137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:21.566963 15137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:21.566995 15137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:21.567009 15137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:21.567022 15137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:21.567034 15137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:21.568373 15137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:21.568552 15137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:21.568702 15137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:21.573298 15143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:21.573308 15142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:21.573370 15137 server_base.cc:1034] running on GCE node
W20260924 05:11:21.573313 15145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:21.573817 15137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:21.574129 15137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:21.575294 15137 hybrid_clock.cc:648] HybridClock initialized: now 1790226681575281 us; error 27 us; skew 500 ppm
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:21.577558 15137 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:11:21.578019 15137 webserver.cc:460] Webserver started at http://127.0.98.62:40727/ using document root <none> and password file <none>
I20260924 05:11:21.578249 15137 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:21.578315 15137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:21.579659 15137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:21.580309 15152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.580471 15137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:21.580570 15137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "327fd3ed012041d8830a50b2e21bc180"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "f0518dc4da732db4a87bf42489b84bee"
server_key_iv: "7c24df4da608a58c7f20a5c09c32c0f3"
server_key_version: "encryptionkey@0"
I20260924 05:11:21.580883 15137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:21.594743 15137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:21.596491 15137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:21.596684 15137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:21.600644 15137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36995
I20260924 05:11:21.600687 15204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36995 every 8 connection(s)
I20260924 05:11:21.601049 15137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:11:21.601533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 15137
I20260924 05:11:21.601939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40981
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36995
--webserver_port=43913
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:21.603390 15205 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:11:21.604071 15205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Bootstrap starting.
I20260924 05:11:21.612274 15205 log.cc:824] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:21.614405 15205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:11:21.614703 15205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180: Bootstrap complete.
I20260924 05:11:21.617131 15205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:21.617352 15205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327fd3ed012041d8830a50b2e21bc180, State: Initialized, Role: FOLLOWER
I20260924 05:11:21.617519 15205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:21.617578 15205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:11:21.617626 15205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:11:21.617673 15205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:11:21.618615 15205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:21.618726 15205 leader_election.cc:302] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 327fd3ed012041d8830a50b2e21bc180; no voters:
I20260924 05:11:21.618907 15205 leader_election.cc:288] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 05:11:21.618940 15209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 05:11:21.619047 15209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [term 3 LEADER]: Becoming Leader. State: Replica: 327fd3ed012041d8830a50b2e21bc180, State: Running, Role: LEADER
I20260924 05:11:21.619127 15209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } }
I20260924 05:11:21.619167 15205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:21.619362 15209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 327fd3ed012041d8830a50b2e21bc180. Latest consensus state: current_term: 3 leader_uuid: "327fd3ed012041d8830a50b2e21bc180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } } }
I20260924 05:11:21.619441 15209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:21.619365 15210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "327fd3ed012041d8830a50b2e21bc180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327fd3ed012041d8830a50b2e21bc180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36995 } } }
I20260924 05:11:21.619639 15210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327fd3ed012041d8830a50b2e21bc180 [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:21.619663 15213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:11:21.620497 15213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:11:21.620609 15213 catalog_manager.cc:1259] Loaded cluster ID: 67437826f29449948c96324a318f43c3
I20260924 05:11:21.620642 15213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 05:11:21.621112 15213 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2403ef29ad google::LogMessage::Fail() at ??:0
@ 0x7f2403ef588a google::LogMessage::SendToLog() at ??:0
@ 0x7f2403ef24de google::LogMessage::Flush() at ??:0
@ 0x7f2403ef3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f240be130f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f240be1c8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2405dc00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2405db5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2406c196db start_thread at ??:0
@ 0x7f24029f061f clone at ??:0
W20260924 05:11:21.711254 15207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:21.711520 15207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:21.711555 15207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:21.713112 15207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:21.713173 15207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:21.713212 15207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:11:21.715346 15207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43913
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:21.715597 15207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:21.715854 15207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:21.721076 15244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:21.721061 15241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:21.721065 15242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:21.721354 15207 server_base.cc:1034] running on GCE node
I20260924 05:11:21.721539 15207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:21.721800 15207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:21.722970 15207 hybrid_clock.cc:648] HybridClock initialized: now 1790226681722956 us; error 26 us; skew 500 ppm
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:21.725809 15207 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:11:21.726212 15207 webserver.cc:460] Webserver started at http://127.0.98.1:43913/ using document root <none> and password file <none>
I20260924 05:11:21.726462 15207 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:21.726536 15207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:21.727927 15207 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:21.728628 15251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.728809 15207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:21.728910 15207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "e35da93437b443ee943b114b6e2245a5"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "98651adbb1d6e5013246c878b526f084"
server_key_iv: "3bfbe7103129eacc7733cdc8a0ac2225"
server_key_version: "encryptionkey@0"
I20260924 05:11:21.729229 15207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:21.743878 15207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:21.746114 15207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:21.746266 15207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:21.746480 15207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:21.747028 15258 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:11:21.748965 15207 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:11:21.749018 15207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:11:21.749059 15207 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:11:21.749632 15207 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:11:21.749704 15207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:11:21.749745 15258 tablet_bootstrap.cc:492] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Bootstrap starting.
I20260924 05:11:21.751575 15258 log.cc:824] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:21.754346 15258 tablet_bootstrap.cc:492] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:11:21.754649 15258 tablet_bootstrap.cc:492] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Bootstrap complete.
I20260924 05:11:21.754822 15258 ts_tablet_manager.cc:1397] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260924 05:11:21.756582 15207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40981
I20260924 05:11:21.756618 15258 raft_consensus.cc:348] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.756961 15258 raft_consensus.cc:729] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35da93437b443ee943b114b6e2245a5, State: Initialized, Role: FOLLOWER
I20260924 05:11:21.756986 15207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:11:21.757139 15258 consensus_queue.cc:260] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:21.757427 15258 ts_tablet_manager.cc:1428] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:11:21.757632 15366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40981 every 8 connection(s)
I20260924 05:11:21.767153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 15207
I20260924 05:11:21.767436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34865
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36995
--webserver_port=46747
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:21.854252 15370 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:11:21.757786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:54501)
0924 05:11:21.757967 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.758004 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:21.758150 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:21.854034 (+ 95884us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
W20260924 05:11:21.854521 15367 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36995 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36995: Client connection negotiation failed: client connection to 127.0.98.62:36995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:11:21.854903 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.854740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:45169)
0924 05:11:21.854826 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.854885 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 05:11:21.855232 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.855155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:39467)
0924 05:11:21.855199 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.855219 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 05:11:21.855350 15367 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:11:21.868355 15371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:21.868537 15371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:21.868561 15371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:21.869635 15371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:21.869668 15371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:21.869694 15371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:11:21.871104 15371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46747
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:21.871284 15371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:21.871467 15371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:21.876327 15376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:21.876317 15377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:21.876363 15371 server_base.cc:1034] running on GCE node
W20260924 05:11:21.876381 15379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:21.876751 15371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:21.876992 15371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:21.878153 15371 hybrid_clock.cc:648] HybridClock initialized: now 1790226681878136 us; error 30 us; skew 500 ppm
Sep 24 05:11:21 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:21.880478 15371 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:11:21.880880 15371 webserver.cc:460] Webserver started at http://127.0.98.2:46747/ using document root <none> and password file <none>
I20260924 05:11:21.881109 15371 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:21.881212 15371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:21.882472 15371 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:21.883136 15386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.883308 15371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.883402 15371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "97e6fca1d7d34656aed551eb9bbccfeb"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "8b8e021d792c9492d21e6a69c0f3f744"
server_key_iv: "dae8a266aa57855ee60e9b0115b2b48a"
server_key_version: "encryptionkey@0"
I20260924 05:11:21.883733 15371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:21.897364 15371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:21.899225 15371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:21.899386 15371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:21.899607 15371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:21.900009 15393 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 05:11:21.901355 15393 ts_tablet_manager.cc:1852] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 05:11:21.901626 15393 ts_tablet_manager.cc:1905] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:21.903540 15393 ts_tablet_manager.cc:1918] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:21.903666 15393 ts_tablet_manager.cc:1939] T 214b8340f4a5491aa87bc3b28692d8b6 P 97e6fca1d7d34656aed551eb9bbccfeb: Deleting consensus metadata
I20260924 05:11:21.903890 15371 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 05:11:21.903949 15371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 05:11:21.904012 15371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:21.904043 15371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:21.909914 15371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34865
I20260924 05:11:21.909997 15500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34865 every 8 connection(s)
I20260924 05:11:21.910284 15371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:11:21.911341 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.910933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:60573)
0924 05:11:21.911080 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.911177 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
W20260924 05:11:21.911511 15501 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36995 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36995: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
I20260924 05:11:21.911779 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.911688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:38805)
0924 05:11:21.911735 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.911759 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:11:21.912072 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:21.911991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:37271)
0924 05:11:21.912033 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:21.912054 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:11:21.912165 15501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:11:21.913151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 15371
I20260924 05:11:21.913343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36211
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36995
--webserver_port=36369
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:21.992393 15504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:21.992609 15504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:21.992645 15504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:21.993742 15504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:21.993803 15504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:21.993846 15504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:11:21.995365 15504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36369
--tserver_master_addrs=127.0.98.62:36995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:21.995617 15504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:21.995894 15504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:22.000747 15512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:22.000721 15510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:22.000733 15509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:22.000913 15504 server_base.cc:1034] running on GCE node
I20260924 05:11:22.001119 15504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:22.001384 15504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:22.002562 15504 hybrid_clock.cc:648] HybridClock initialized: now 1790226682002544 us; error 30 us; skew 500 ppm
Sep 24 05:11:22 dist-test-slave-mr2f krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:22.004987 15504 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:11:22.005407 15504 webserver.cc:460] Webserver started at http://127.0.98.3:36369/ using document root <none> and password file <none>
I20260924 05:11:22.005652 15504 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:22.005722 15504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:22.007011 15504 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:11:22.007603 15519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:22.007764 15504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:22.007875 15504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "5c7f7d45eec841aab64f64ecffe2c80e"
format_stamp: "Formatted at 2026-09-24 05:11:20 on dist-test-slave-mr2f"
server_key: "d37c44ff938b5464bf551f3a64e59128"
server_key_iv: "3bda014803b09eb8e55e2d46bb05b77e"
server_key_version: "encryptionkey@0"
I20260924 05:11:22.008190 15504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:22.023768 15504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:22.025341 15504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:22.025494 15504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:22.025700 15504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:22.025998 15504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:22.026052 15504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:22.026090 15504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:22.026118 15504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:22.031733 15504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36211
I20260924 05:11:22.031751 15632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36211 every 8 connection(s)
I20260924 05:11:22.032164 15504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:11:22.033210 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:22.032819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:52917)
0924 05:11:22.032986 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:22.033082 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
W20260924 05:11:22.033600 15633 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36995 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36995: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
I20260924 05:11:22.033835 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:22.033765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:46163)
0924 05:11:22.033793 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:22.033818 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 05:11:22.034109 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:22.034023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:57659)
0924 05:11:22.034067 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:22.034096 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:11:22.034225 15633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:11:22.037953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 15504
I20260924 05:11:22.855835 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:22.855637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:54885)
0924 05:11:22.855746 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:22.855778 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260924 05:11:22.912637 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:22.912449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:49207)
0924 05:11:22.912557 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:22.912594 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260924 05:11:23.034647 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:23.034480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:60177)
0924 05:11:23.034578 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:23.034607 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260924 05:11:23.229990 15640 raft_consensus.cc:482] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:23.230163 15640 raft_consensus.cc:504] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:23.230523 15640 leader_election.cc:288] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865), 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211)
Sep 24 05:11:23 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 05:11:23 dist-test-slave-mr2f krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 05:11:23.240980 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:23.231015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50975 (local address 127.0.98.3:36211)
0924 05:11:23.231168 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:23.231176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:23.231202 (+ 26us) server_negotiation.cc:407] Connection header received
0924 05:11:23.231639 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:23.231642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:23.231691 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:23.231831 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:23.232509 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:23.233475 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:23.234486 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:23.234912 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:23.235038 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:23.237214 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:23.237219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:23.237219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:23.237230 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:23.239180 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:23.239640 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:23.239642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:23.239643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:23.239760 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:23.240043 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:23.240043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:23.240044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:23.240324 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:23.240410 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:23.240760 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 05:11:23.240878 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 05:11:23.240980 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:23.230791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40339 (local address 127.0.98.2:34865)
0924 05:11:23.230990 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:23.231048 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:23.231074 (+ 26us) server_negotiation.cc:407] Connection header received
0924 05:11:23.231607 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:23.231610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:23.231670 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:23.231831 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:23.232630 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:23.233514 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:23.234478 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:23.234913 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:23.235045 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:23.237578 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:23.237581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:23.237581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:23.237590 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:23.239370 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:23.239726 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:23.239728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:23.239728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:23.239822 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:23.240043 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:23.240043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:23.240044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:23.240337 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:23.240410 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:23.240760 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 05:11:23.240878 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260924 05:11:23.240983 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:23.230684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34865 (local address 127.0.98.1:40339)
0924 05:11:23.230806 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:23.230818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:23.230875 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:23.231834 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:23.231836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:23.232455 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:23.232527 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:23.232529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:23.233618 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:23.233619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:23.234392 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:23.234394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:23.235010 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:23.235011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:23.235121 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:23.235728 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:23.235732 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:23.237477 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:23.239441 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:23.239442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:23.239443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:23.239660 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:23.239887 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:23.239888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:23.239888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:23.239969 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:23.240405 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:23.240411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:23.240558 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 05:11:23.240692 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 05:11:23.240869 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":62}
W20260924 05:11:23.241739 15254 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
I20260924 05:11:23.241767 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:23.230809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36211 (local address 127.0.98.1:50975)
0924 05:11:23.231010 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:23.231018 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:23.231057 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:23.231840 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:23.231841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:23.232188 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:23.232395 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:23.232400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:23.233585 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:23.233586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:23.234394 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:23.234396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:23.235011 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:23.235011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:23.235129 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:23.235676 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:23.235683 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:23.237095 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:23.239255 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:23.239260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:23.239262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:23.239559 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:23.239829 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:23.239830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:23.239830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:23.239969 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:23.240405 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:23.240411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:23.240557 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 05:11:23.240692 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 05:11:23.240917 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":123,"thread_start_us":61,"threads_started":1}
W20260924 05:11:23.242398 15255 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
I20260924 05:11:23.242455 15255 leader_election.cc:302] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e35da93437b443ee943b114b6e2245a5; no voters: 5c7f7d45eec841aab64f64ecffe2c80e, 97e6fca1d7d34656aed551eb9bbccfeb
I20260924 05:11:23.242551 15640 raft_consensus.cc:2726] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:11:23.856573 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:23.856332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:38623)
0924 05:11:23.856460 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:23.856517 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260924 05:11:23.913218 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:23.913037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:40849)
0924 05:11:23.913163 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:23.913188 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260924 05:11:24.035305 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:24.035120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:52041)
0924 05:11:24.035237 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:24.035270 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260924 05:11:24.758374 15647 raft_consensus.cc:482] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:24.758447 15647 raft_consensus.cc:504] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:24.758589 15647 leader_election.cc:288] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865), 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211)
W20260924 05:11:24.758987 15254 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
W20260924 05:11:24.759065 15255 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
I20260924 05:11:24.759176 15255 leader_election.cc:302] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e35da93437b443ee943b114b6e2245a5; no voters: 5c7f7d45eec841aab64f64ecffe2c80e, 97e6fca1d7d34656aed551eb9bbccfeb
I20260924 05:11:24.759258 15647 raft_consensus.cc:2726] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:11:24.857235 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:24.856999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:49271)
0924 05:11:24.857152 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:24.857188 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260924 05:11:24.913866 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:24.913686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:34205)
0924 05:11:24.913804 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:24.913833 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260924 05:11:25.035974 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:25.035657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:59881)
0924 05:11:25.035901 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:25.035938 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
I20260924 05:11:25.857905 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:25.857677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:41175)
0924 05:11:25.857814 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:25.857854 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260924 05:11:25.914481 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:25.914292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:53501)
0924 05:11:25.914400 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:25.914436 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260924 05:11:26.036690 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:26.036462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:35117)
0924 05:11:26.036617 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:26.036656 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260924 05:11:26.858548 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:26.858336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:42965)
0924 05:11:26.858469 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:26.858503 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260924 05:11:26.915112 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:26.914939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:55805)
0924 05:11:26.915049 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:26.915080 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260924 05:11:27.037262 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:27.037073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:54611)
0924 05:11:27.037196 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:27.037228 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260924 05:11:27.695020 15657 raft_consensus.cc:482] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:27.695108 15657 raft_consensus.cc:504] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:27.695333 15657 leader_election.cc:288] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865), 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211)
W20260924 05:11:27.695709 15255 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
W20260924 05:11:27.695812 15254 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
I20260924 05:11:27.695850 15254 leader_election.cc:302] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e35da93437b443ee943b114b6e2245a5; no voters: 5c7f7d45eec841aab64f64ecffe2c80e, 97e6fca1d7d34656aed551eb9bbccfeb
I20260924 05:11:27.695945 15657 raft_consensus.cc:2726] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:11:27.859341 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:27.859100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:34823)
0924 05:11:27.859244 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:27.859295 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260924 05:11:27.915725 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:27.915504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:39593)
0924 05:11:27.915631 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:27.915665 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260924 05:11:28.037899 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:28.037701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:39223)
0924 05:11:28.037826 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:28.037860 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260924 05:11:28.859997 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:28.859771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:54623)
0924 05:11:28.859918 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:28.859954 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260924 05:11:28.916289 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:28.916113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:35469)
0924 05:11:28.916225 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:28.916254 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
I20260924 05:11:29.038576 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:29.038325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:48095)
0924 05:11:29.038470 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:29.038510 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260924 05:11:29.860761 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:29.860464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:36101)
0924 05:11:29.860679 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:29.860716 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260924 05:11:29.916895 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:29.916714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:52161)
0924 05:11:29.916829 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:29.916857 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260924 05:11:30.039255 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:30.039046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:47633)
0924 05:11:30.039174 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:30.039208 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260924 05:11:30.861512 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:30.861315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:51933)
0924 05:11:30.861432 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:30.861468 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260924 05:11:30.917490 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:30.917311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:58913)
0924 05:11:30.917421 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:30.917454 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
I20260924 05:11:31.039819 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:31.039637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:55411)
0924 05:11:31.039748 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:31.039779 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260924 05:11:31.862214 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:31.861997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:46049)
0924 05:11:31.862122 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:31.862157 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260924 05:11:31.918119 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:31.917926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:52781)
0924 05:11:31.918043 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:31.918079 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260924 05:11:32.040412 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:32.040231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:45319)
0924 05:11:32.040346 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:32.040379 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260924 05:11:32.044273 15673 raft_consensus.cc:482] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:32.044348 15673 raft_consensus.cc:504] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:32.044564 15673 leader_election.cc:288] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865), 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211)
W20260924 05:11:32.044866 15255 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
W20260924 05:11:32.044963 15254 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
I20260924 05:11:32.045002 15254 leader_election.cc:302] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e35da93437b443ee943b114b6e2245a5; no voters: 5c7f7d45eec841aab64f64ecffe2c80e, 97e6fca1d7d34656aed551eb9bbccfeb
I20260924 05:11:32.045061 15673 raft_consensus.cc:2726] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:11:32.862923 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:32.862659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:54541)
0924 05:11:32.862842 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:32.862876 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":60,"threads_started":1}
I20260924 05:11:32.918824 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:32.918632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:59927)
0924 05:11:32.918737 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:32.918785 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260924 05:11:33.041085 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:33.040901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:46723)
0924 05:11:33.041014 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:33.041046 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260924 05:11:33.863627 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:33.863389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:60351)
0924 05:11:33.863545 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:33.863583 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":54,"threads_started":1}
I20260924 05:11:33.919400 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:33.919235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:59051)
0924 05:11:33.919331 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:33.919362 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":61,"threads_started":1}
I20260924 05:11:34.041648 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:34.041469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:40343)
0924 05:11:34.041584 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:34.041614 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 05:11:34.252791 15680 raft_consensus.cc:482] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:34.252892 15680 raft_consensus.cc:504] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97e6fca1d7d34656aed551eb9bbccfeb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34865 } } peers { permanent_uuid: "5c7f7d45eec841aab64f64ecffe2c80e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36211 } } peers { permanent_uuid: "e35da93437b443ee943b114b6e2245a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40981 } }
I20260924 05:11:34.253098 15680 leader_election.cc:288] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865), 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211)
W20260924 05:11:34.253460 15255 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c7f7d45eec841aab64f64ecffe2c80e (127.0.98.3:36211): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
W20260924 05:11:34.253568 15254 leader_election.cc:341] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 97e6fca1d7d34656aed551eb9bbccfeb (127.0.98.2:34865): Not found: Tablet not found: 214b8340f4a5491aa87bc3b28692d8b6
I20260924 05:11:34.253599 15254 leader_election.cc:302] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e35da93437b443ee943b114b6e2245a5; no voters: 5c7f7d45eec841aab64f64ecffe2c80e, 97e6fca1d7d34656aed551eb9bbccfeb
I20260924 05:11:34.253682 15680 raft_consensus.cc:2726] T 214b8340f4a5491aa87bc3b28692d8b6 P e35da93437b443ee943b114b6e2245a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:11:34.864352 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:34.864072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:34189)
0924 05:11:34.864242 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:34.864296 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
I20260924 05:11:34.919947 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:34.919760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:59499)
0924 05:11:34.919881 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:34.919912 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260924 05:11:35.042217 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:35.042045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:38229)
0924 05:11:35.042148 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:35.042181 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260924 05:11:35.865059 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:35.864851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:52253)
0924 05:11:35.864977 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:35.865014 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260924 05:11:35.920585 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:35.920390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:44619)
0924 05:11:35.920521 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:35.920549 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260924 05:11:36.042860 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:36.042632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:40077)
0924 05:11:36.042751 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:36.042824 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260924 05:11:36.865774 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:36.865580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.1:45183)
0924 05:11:36.865698 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:36.865731 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260924 05:11:36.921372 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:36.921204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.2:46457)
0924 05:11:36.921309 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:36.921335 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260924 05:11:37.043504 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:37.043325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36995 (local address 127.0.98.3:37623)
0924 05:11:37.043435 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:37.043468 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":50,"threads_started":1}
I20260924 05:11:37.043979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 15207
I20260924 05:11:37.048729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 15371
I20260924 05:11:37.052762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 15504
2026-09-24T05:11:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18086 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:11:37 dist-test-slave-mr2f krb5kdc[15692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:11:37 dist-test-slave-mr2f krb5kdc[15692](info): set up 2 sockets
Sep 24 05:11:37 dist-test-slave-mr2f krb5kdc[15692](info): commencing operation
krb5kdc: starting...
W20260924 05:11:38.058818 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:11:38 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:11:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:11:38.169425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:38.248919 15705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:38.249138 15705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:38.249176 15705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:38.250371 15705 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 05:11:38.250424 15705 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 05:11:38.250445 15705 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 05:11:38.250463 15705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:38.250494 15705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:38.250519 15705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:38.250547 15705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:38.250577 15705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:38.252108 15705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:38.252367 15705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:38.252610 15705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:11:38.257613 15705 server_base.cc:1034] running on GCE node
W20260924 05:11:38.257680 15713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:38.257719 15710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:38.257668 15711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:38.258068 15705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:38.258368 15705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:38.259508 15705 hybrid_clock.cc:648] HybridClock initialized: now 1790226698259507 us; error 39 us; skew 500 ppm
Sep 24 05:11:38 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:38.268146 15705 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:11:38.268613 15705 webserver.cc:460] Webserver started at http://127.0.98.62:39649/ using document root <none> and password file <none>
I20260924 05:11:38.268833 15705 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:38.268909 15705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:38.269052 15705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:38.270053 15705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "7437708936eb43faacc08fcd15e9e2fd"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "d0c7023b8d0f4b0135a55b32ad563c91"
server_key_iv: "12110b6a80c36dfc51c5fc9340bd3a03"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.270406 15705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "7437708936eb43faacc08fcd15e9e2fd"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "d0c7023b8d0f4b0135a55b32ad563c91"
server_key_iv: "12110b6a80c36dfc51c5fc9340bd3a03"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.271752 15705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:11:38.272469 15720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.272639 15705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:11:38.272742 15705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "7437708936eb43faacc08fcd15e9e2fd"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "d0c7023b8d0f4b0135a55b32ad563c91"
server_key_iv: "12110b6a80c36dfc51c5fc9340bd3a03"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.272812 15705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:38.283821 15705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:38.285544 15705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:38.285709 15705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:38.289741 15705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44387
I20260924 05:11:38.289796 15772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44387 every 8 connection(s)
I20260924 05:11:38.290143 15705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:11:38.291013 15773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:38.293140 15773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Bootstrap starting.
I20260924 05:11:38.293718 15773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:38.293742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 15705
I20260924 05:11:38.293879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:11:38.294032 15773 log.cc:824] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:38.294066 392 external_mini_cluster.cc:1428] Setting key faed2811a725612b1f8f7118877c16bb
I20260924 05:11:38.294821 15773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: No bootstrap required, opened a new log
I20260924 05:11:38.297331 15773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:11:38.297487 15773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:38.297519 15773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7437708936eb43faacc08fcd15e9e2fd, State: Initialized, Role: FOLLOWER
I20260924 05:11:38.297646 15773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:11:38.297732 15773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:11:38.297766 15773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:11:38.297820 15773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:38.298369 15773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:11:38.298470 15773 leader_election.cc:302] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7437708936eb43faacc08fcd15e9e2fd; no voters:
I20260924 05:11:38.298626 15773 leader_election.cc:288] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:11:38.298691 15778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:38.298909 15778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 1 LEADER]: Becoming Leader. State: Replica: 7437708936eb43faacc08fcd15e9e2fd, State: Running, Role: LEADER
I20260924 05:11:38.298924 15773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:38.299052 15778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:11:38.299423 15779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7437708936eb43faacc08fcd15e9e2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } }
I20260924 05:11:38.299436 15780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7437708936eb43faacc08fcd15e9e2fd. Latest consensus state: current_term: 1 leader_uuid: "7437708936eb43faacc08fcd15e9e2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } }
I20260924 05:11:38.299556 15780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:38.299525 15779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:38.299868 15784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:11:38.300408 15784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:11:38 dist-test-slave-mr2f krb5kdc[15692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:38.302045 15784 catalog_manager.cc:1347] Generated new cluster ID: 3734350cabd74946b9240aa877727cf2
I20260924 05:11:38.302106 15784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:11:38.305531 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.294540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.0.98.62:44387)
0924 05:11:38.294732 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.294741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.294786 (+ 45us) server_negotiation.cc:407] Connection header received
0924 05:11:38.295281 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.295289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.295348 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.295493 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:38.296074 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.297511 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.298371 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.299050 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.299158 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.302023 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:38.302027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:38.302027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:38.302037 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:38.303884 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.304332 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.304335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.304335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.304428 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.304642 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.304643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.304643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.304901 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:38.305006 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.305187 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.305312 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260924 05:11:38.313058 15784 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:11:38.313524 15784 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:11:38.315852 15784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Generated new TSK 0
I20260924 05:11:38.315996 15784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:11:38.341372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44387
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:38.419685 15799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:38.419906 15799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:38.419945 15799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:38.421063 15799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:38.421136 15799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:38.421180 15799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:11:38.422600 15799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:38.422875 15799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:38.423094 15799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:38.427820 15804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:38.427826 15807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:38.427850 15805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:38.428042 15799 server_base.cc:1034] running on GCE node
I20260924 05:11:38.428236 15799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:38.428488 15799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:38.429634 15799 hybrid_clock.cc:648] HybridClock initialized: now 1790226698429628 us; error 35 us; skew 500 ppm
Sep 24 05:11:38 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:38.431988 15799 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:11:38.432436 15799 webserver.cc:460] Webserver started at http://127.0.98.1:43349/ using document root <none> and password file <none>
I20260924 05:11:38.432653 15799 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:38.432716 15799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:38.432858 15799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:38.433871 15799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "c76a779267cb4195aecd226588577e6a"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "9735f85daf523b5edf1372a3b9bcc8c4"
server_key_iv: "b49bf7720d8227e26080381993e8e5f1"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.434224 15799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c76a779267cb4195aecd226588577e6a"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "9735f85daf523b5edf1372a3b9bcc8c4"
server_key_iv: "b49bf7720d8227e26080381993e8e5f1"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.435541 15799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:11:38.436206 15814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.436360 15799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:11:38.436481 15799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "c76a779267cb4195aecd226588577e6a"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "9735f85daf523b5edf1372a3b9bcc8c4"
server_key_iv: "b49bf7720d8227e26080381993e8e5f1"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.436563 15799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:38.462491 15799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:38.464288 15799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:38.464445 15799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:38.464671 15799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:38.464962 15799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:38.465013 15799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.465054 15799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:38.465081 15799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.470944 15799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35615
I20260924 05:11:38.470997 15927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35615 every 8 connection(s)
I20260924 05:11:38.471297 15799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:11:38.476037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 15799
I20260924 05:11:38.476169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:11:38.476398 392 external_mini_cluster.cc:1428] Setting key bd1fd27785781174f53958899396e2ee
Sep 24 05:11:38 dist-test-slave-mr2f krb5kdc[15692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:38.480685 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.472003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57293 (local address 127.0.98.62:44387)
0924 05:11:38.472050 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.472052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.472261 (+ 209us) server_negotiation.cc:407] Connection header received
0924 05:11:38.472684 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.472685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.472697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.472755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:38.473621 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.474115 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.475113 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.475316 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.475368 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.477810 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:38.477812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:38.477812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:38.477819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:38.479321 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.479710 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.479711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.479711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.479775 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.480008 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.480010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.480010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.480156 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:38.480225 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.480455 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.480547 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:11:38.480790 15930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.471926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:57293)
0924 05:11:38.472096 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.472146 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.472602 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.472810 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.472814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.473356 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:38.473538 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.473542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.474195 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.474196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.474990 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.474992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.475387 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.475388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.475483 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.475956 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:38.475966 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:38.477711 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:38.479409 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:38.479414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:38.479415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:38.479635 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:38.479843 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:38.479843 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:38.479843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:38.479935 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:38.480218 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:38.480222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:38.480326 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.480402 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.480602 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260924 05:11:38.481225 15928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44387
I20260924 05:11:38.481310 15928 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:38.481487 15928 heartbeater.cc:511] Master 127.0.98.62:44387 requested a full tablet report, sending...
I20260924 05:11:38.482043 15737 ts_manager.cc:194] Registered new tserver with Master: c76a779267cb4195aecd226588577e6a (127.0.98.1:35615)
I20260924 05:11:38.482964 15737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57293
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:11:38.499029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44387
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:38.576738 15933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:38.576931 15933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:38.576954 15933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:38.578105 15933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:38.578148 15933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:38.578212 15933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:11:38.579662 15933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:38.579888 15933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:38.580107 15933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:38.585036 15941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:38.585121 15933 server_base.cc:1034] running on GCE node
W20260924 05:11:38.585209 15938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:38.585096 15939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:38.585469 15933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:38.585733 15933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:38.586889 15933 hybrid_clock.cc:648] HybridClock initialized: now 1790226698586872 us; error 47 us; skew 500 ppm
Sep 24 05:11:38 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:38.589202 15933 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:11:38.589648 15933 webserver.cc:460] Webserver started at http://127.0.98.2:42367/ using document root <none> and password file <none>
I20260924 05:11:38.589874 15933 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:38.589938 15933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:38.590082 15933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:38.591177 15933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "92f8e028c1b1465481daa6a711279530"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "dd744943439f806dc28171ff7e2df101"
server_key_iv: "6389a71752401368565218c532426d9e"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.591522 15933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "92f8e028c1b1465481daa6a711279530"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "dd744943439f806dc28171ff7e2df101"
server_key_iv: "6389a71752401368565218c532426d9e"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.592839 15933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:11:38.593519 15948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.593686 15933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:38.593780 15933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "92f8e028c1b1465481daa6a711279530"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "dd744943439f806dc28171ff7e2df101"
server_key_iv: "6389a71752401368565218c532426d9e"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.593859 15933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:38.603699 15933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:38.605363 15933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:38.605522 15933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:38.605748 15933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:38.606037 15933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:38.606091 15933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.606128 15933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:38.606153 15933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.611681 15933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43717
I20260924 05:11:38.611738 16061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43717 every 8 connection(s)
I20260924 05:11:38.612079 15933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:11:38.613165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 15933
I20260924 05:11:38.613258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:11:38.613479 392 external_mini_cluster.cc:1428] Setting key f75e636969b5aa47e8ab5bd55407db2b
Sep 24 05:11:38 dist-test-slave-mr2f krb5kdc[15692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 05:11:38.622113 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.612826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50309 (local address 127.0.98.62:44387)
0924 05:11:38.612907 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.612909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.613110 (+ 201us) server_negotiation.cc:407] Connection header received
0924 05:11:38.613862 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.613864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.613875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.613929 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:38.615184 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.615799 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.616786 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.617006 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.617063 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.619443 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:38.619444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:38.619444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:38.619451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:38.620791 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.621187 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.621189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.621189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.621245 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.621487 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.621488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.621489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.621597 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:38.621674 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.621947 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.622027 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 05:11:38.622291 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.612786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:50309)
0924 05:11:38.612954 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.613016 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.613745 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.613990 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.613993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.614818 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:38.615086 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.615092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.615898 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.615899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.616709 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.616711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.617097 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.617098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.617243 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.617739 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:38.617749 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:38.619340 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:38.620881 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:38.620886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:38.620888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:38.621099 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:38.621318 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:38.621319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:38.621319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:38.621422 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:38.621664 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:38.621670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:38.621803 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.621917 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.622061 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260924 05:11:38.622823 16062 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44387
I20260924 05:11:38.622911 16062 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:38.623049 16062 heartbeater.cc:511] Master 127.0.98.62:44387 requested a full tablet report, sending...
I20260924 05:11:38.623399 15737 ts_manager.cc:194] Registered new tserver with Master: 92f8e028c1b1465481daa6a711279530 (127.0.98.2:43717)
I20260924 05:11:38.623836 15737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:11:38.637467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44387
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:38.716866 16067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:38.717100 16067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:38.717139 16067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:38.718279 16067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:38.718348 16067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:38.718393 16067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:11:38.719851 16067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:38.720103 16067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:38.720317 16067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:38.725133 16075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:38.725145 16072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:38.725167 16073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:38.725350 16067 server_base.cc:1034] running on GCE node
I20260924 05:11:38.725569 16067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:38.725847 16067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:38.727000 16067 hybrid_clock.cc:648] HybridClock initialized: now 1790226698726994 us; error 34 us; skew 500 ppm
Sep 24 05:11:38 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:38.729331 16067 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:11:38.729766 16067 webserver.cc:460] Webserver started at http://127.0.98.3:40623/ using document root <none> and password file <none>
I20260924 05:11:38.729974 16067 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:38.730036 16067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:38.730180 16067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:11:38.731192 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4e7708d96294878913f7b0ff6e6a0ee"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "e919127dbbbcfa06cc03f7f0a5c3ccb8"
server_key_iv: "fa9c0b96a6c3106edcbbac950624542a"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.731527 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4e7708d96294878913f7b0ff6e6a0ee"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "e919127dbbbcfa06cc03f7f0a5c3ccb8"
server_key_iv: "fa9c0b96a6c3106edcbbac950624542a"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.732805 16067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:11:38.733436 16082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.733588 16067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:38.733678 16067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "e4e7708d96294878913f7b0ff6e6a0ee"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "e919127dbbbcfa06cc03f7f0a5c3ccb8"
server_key_iv: "fa9c0b96a6c3106edcbbac950624542a"
server_key_version: "encryptionkey@0"
I20260924 05:11:38.733753 16067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:38.745111 16067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:38.747303 16067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:38.747453 16067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:38.747681 16067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:38.747963 16067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:38.748015 16067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.748049 16067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:38.748076 16067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:38.753804 16067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41727
I20260924 05:11:38.753851 16195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41727 every 8 connection(s)
I20260924 05:11:38.754146 16067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:11:38 dist-test-slave-mr2f krb5kdc[15692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:38.762236 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.754869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33277 (local address 127.0.98.62:44387)
0924 05:11:38.754925 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.754927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.755087 (+ 160us) server_negotiation.cc:407] Connection header received
0924 05:11:38.755585 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.755585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.755595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.755657 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:38.756500 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.756976 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.757886 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.758054 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.758103 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.760096 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:38.760097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:38.760097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:38.760116 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:38.761212 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.761538 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.761538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.761539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.761583 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.761775 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.761776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.761776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.761862 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:38.761909 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.762114 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.762175 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:11:38.762313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 16067
I20260924 05:11:38.762408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:11:38.762485 16198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.754829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:33277)
0924 05:11:38.754962 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.755002 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.755479 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.755715 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.755718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.756230 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:38.756420 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.756425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.757053 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.757054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.757816 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.757818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.758121 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.758122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.758212 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.758668 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:38.758676 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:38.760014 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:38.761279 (+ 1265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:38.761284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:38.761285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:38.761478 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:38.761639 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:38.761639 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:38.761640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:38.761725 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:38.761914 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:38.761917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:38.762012 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.762088 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.762253 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260924 05:11:38.762640 392 external_mini_cluster.cc:1428] Setting key c33338579196d02ce629ddda8fe9e692
I20260924 05:11:38.763214 16196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44387
I20260924 05:11:38.763327 16196 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:38.763491 16196 heartbeater.cc:511] Master 127.0.98.62:44387 requested a full tablet report, sending...
I20260924 05:11:38.763842 15736 ts_manager.cc:194] Registered new tserver with Master: e4e7708d96294878913f7b0ff6e6a0ee (127.0.98.3:41727)
I20260924 05:11:38.764211 15736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33277
I20260924 05:11:38.774309 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:11:38.781180 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.775718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.0.98.62:44387)
0924 05:11:38.775760 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.775762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.775865 (+ 103us) server_negotiation.cc:407] Connection header received
0924 05:11:38.775902 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.775903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.775911 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.775961 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:38.776471 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.776982 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.777678 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.777845 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.777893 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.779151 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:38.779151 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:38.779152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:38.779157 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:38.780237 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.780521 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.780522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.780522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.780569 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:38.780773 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:38.780773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:38.780773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:38.780882 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:38.780949 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.781035 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.781087 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:11:38.782946 15736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:11:38.784375 15736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:11:38.790369 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.786300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.0.1:52070)
0924 05:11:38.786485 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.786525 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.786588 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.786897 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.786899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.786901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:38.786961 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.786963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.787929 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.787930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.789145 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.789147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.790115 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.790116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.790176 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.790181 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.790226 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.790294 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":52,"threads_started":1}
I20260924 05:11:38.790397 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.786621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.0.98.1:35615)
0924 05:11:38.786826 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.786831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.786842 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:11:38.786886 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.786888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.786903 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.786976 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:38.787084 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.788171 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.789380 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.789941 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.790037 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.790092 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.790271 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.790317 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
I20260924 05:11:38.790473 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.786560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.0.98.3:41727)
0924 05:11:38.787130 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.787135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.787144 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:11:38.787182 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.787183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.787197 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.787397 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:38.787461 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.788417 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.789492 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.790146 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.790210 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.790259 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.790343 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.790380 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":89,"threads_started":1}
I20260924 05:11:38.790473 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.786471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41727 (local address 127.0.0.1:40818)
0924 05:11:38.786852 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.786863 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.786912 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.787293 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.787294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.787296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:38.787334 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.787335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.788507 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.788508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.789412 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.789415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.790220 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.790221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.790282 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.790285 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.790334 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.790370 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":66,"threads_started":1}
I20260924 05:11:38.790815 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.786300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35615 (local address 127.0.0.1:36592)
0924 05:11:38.786679 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.786688 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.786725 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.786986 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.786987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.786989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:38.787030 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.787031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.788276 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.788277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.789304 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.789306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.790039 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.790040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.790118 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.790138 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.790227 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.790706 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":437,"thread_start_us":49,"threads_started":1}
I20260924 05:11:38.790868 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.786395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.0.98.2:43717)
0924 05:11:38.786702 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.786706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.786717 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:11:38.786753 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.786755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.786806 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.786880 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:38.787049 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.787829 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.789232 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.789962 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.790027 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.790081 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.790706 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.790755 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":105,"threads_started":1}
I20260924 05:11:38.791440 15862 tablet_service.cc:1467] Processing CreateTablet for tablet 370852666c8a46c795fad36bbee7ba93 (DEFAULT_TABLE table=test-table [id=192b76313fbc4c4dbf1593b7ea1142df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:38.791464 16130 tablet_service.cc:1467] Processing CreateTablet for tablet 370852666c8a46c795fad36bbee7ba93 (DEFAULT_TABLE table=test-table [id=192b76313fbc4c4dbf1593b7ea1142df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:38.791464 15996 tablet_service.cc:1467] Processing CreateTablet for tablet 370852666c8a46c795fad36bbee7ba93 (DEFAULT_TABLE table=test-table [id=192b76313fbc4c4dbf1593b7ea1142df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:38.791757 15862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370852666c8a46c795fad36bbee7ba93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:38.791761 16130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370852666c8a46c795fad36bbee7ba93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:38.791761 15996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370852666c8a46c795fad36bbee7ba93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:38.794054 16210 tablet_bootstrap.cc:492] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Bootstrap starting.
I20260924 05:11:38.794865 16211 tablet_bootstrap.cc:492] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: Bootstrap starting.
I20260924 05:11:38.795034 16210 tablet_bootstrap.cc:654] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:38.795542 16211 tablet_bootstrap.cc:654] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:38.795614 16210 log.cc:824] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:38.796012 16212 tablet_bootstrap.cc:492] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Bootstrap starting.
I20260924 05:11:38.796110 16211 log.cc:824] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:38.796674 16212 tablet_bootstrap.cc:654] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:38.796854 16211 tablet_bootstrap.cc:492] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: No bootstrap required, opened a new log
I20260924 05:11:38.796934 16211 ts_tablet_manager.cc:1397] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:11:38.797209 16212 log.cc:824] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:38.797731 16210 tablet_bootstrap.cc:492] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: No bootstrap required, opened a new log
I20260924 05:11:38.797802 16210 ts_tablet_manager.cc:1397] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 05:11:38.799068 16212 tablet_bootstrap.cc:492] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: No bootstrap required, opened a new log
I20260924 05:11:38.799016 16211 raft_consensus.cc:348] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.799156 16212 ts_tablet_manager.cc:1397] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 05:11:38.799201 16211 raft_consensus.cc:374] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:38.799266 16211 raft_consensus.cc:729] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e7708d96294878913f7b0ff6e6a0ee, State: Initialized, Role: FOLLOWER
I20260924 05:11:38.799381 16211 consensus_queue.cc:260] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.799753 16211 ts_tablet_manager.cc:1428] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:11:38.799814 16196 heartbeater.cc:503] Master 127.0.98.62:44387 was elected leader, sending a full tablet report...
I20260924 05:11:38.799856 16210 raft_consensus.cc:348] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.800004 16210 raft_consensus.cc:374] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:38.800035 16210 raft_consensus.cc:729] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76a779267cb4195aecd226588577e6a, State: Initialized, Role: FOLLOWER
I20260924 05:11:38.800159 16210 consensus_queue.cc:260] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.800401 16210 ts_tablet_manager.cc:1428] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 05:11:38.800432 15928 heartbeater.cc:503] Master 127.0.98.62:44387 was elected leader, sending a full tablet report...
I20260924 05:11:38.801366 16212 raft_consensus.cc:348] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.801527 16212 raft_consensus.cc:374] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:38.801571 16212 raft_consensus.cc:729] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f8e028c1b1465481daa6a711279530, State: Initialized, Role: FOLLOWER
I20260924 05:11:38.801677 16212 consensus_queue.cc:260] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.801918 16212 ts_tablet_manager.cc:1428] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:11:38.801954 16062 heartbeater.cc:503] Master 127.0.98.62:44387 was elected leader, sending a full tablet report...
W20260924 05:11:38.862653 16063 tablet.cc:2369] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:11:38.917404 16218 raft_consensus.cc:482] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:38.917536 16218 raft_consensus.cc:504] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.917873 16218 leader_election.cc:288] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4e7708d96294878913f7b0ff6e6a0ee (127.0.98.3:41727), c76a779267cb4195aecd226588577e6a (127.0.98.1:35615)
I20260924 05:11:38.920818 16216 raft_consensus.cc:482] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:38.920905 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.918190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53345 (local address 127.0.98.3:41727)
0924 05:11:38.918233 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.918235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.918245 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:11:38.918284 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.918285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.918295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.918381 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:38.918599 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.919162 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.920097 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.920547 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.920599 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.920607 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.920804 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.920847 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:11:38.920894 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.918079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41727 (local address 127.0.98.2:53345)
0924 05:11:38.918124 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.918136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.918179 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.918407 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.918408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.918410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:38.918504 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.918506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.919239 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.919240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.920005 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.920007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.920623 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.920624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.920701 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.920704 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.920767 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.920804 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:11:38.920926 16216 raft_consensus.cc:504] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.921270 16216 leader_election.cc:288] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f8e028c1b1465481daa6a711279530 (127.0.98.2:43717), c76a779267cb4195aecd226588577e6a (127.0.98.1:35615)
I20260924 05:11:38.921255 16150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370852666c8a46c795fad36bbee7ba93" candidate_uuid: "92f8e028c1b1465481daa6a711279530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" is_pre_election: true
I20260924 05:11:38.921398 16150 raft_consensus.cc:2454] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f8e028c1b1465481daa6a711279530 in term 0.
I20260924 05:11:38.921557 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.918166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50509 (local address 127.0.98.1:35615)
0924 05:11:38.918217 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.918219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.918485 (+ 266us) server_negotiation.cc:407] Connection header received
0924 05:11:38.918514 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.918516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.918541 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.918600 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:38.918806 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.919302 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.920345 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.920811 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.920871 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.920879 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.921402 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.921465 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:11:38.921780 15949 leader_election.cc:302] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f8e028c1b1465481daa6a711279530, e4e7708d96294878913f7b0ff6e6a0ee; no voters:
I20260924 05:11:38.921908 16218 raft_consensus.cc:2781] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:11:38.921957 16218 raft_consensus.cc:482] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:11:38.921981 16218 raft_consensus.cc:3037] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:38.921314 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.918079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35615 (local address 127.0.98.2:50509)
0924 05:11:38.918408 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.918416 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.918465 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.918630 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.918631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.918633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:38.918697 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.918698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.919380 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.919381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.920250 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.920253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.921082 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.921083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.921148 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.921152 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.921197 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.921230 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":149,"thread_start_us":120,"threads_started":1}
I20260924 05:11:38.922932 16218 raft_consensus.cc:504] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.923237 16150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370852666c8a46c795fad36bbee7ba93" candidate_uuid: "92f8e028c1b1465481daa6a711279530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4e7708d96294878913f7b0ff6e6a0ee"
I20260924 05:11:38.923267 15882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370852666c8a46c795fad36bbee7ba93" candidate_uuid: "92f8e028c1b1465481daa6a711279530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c76a779267cb4195aecd226588577e6a" is_pre_election: true
I20260924 05:11:38.923316 16150 raft_consensus.cc:3037] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:38.923403 15882 raft_consensus.cc:2454] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f8e028c1b1465481daa6a711279530 in term 0.
I20260924 05:11:38.923784 15882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370852666c8a46c795fad36bbee7ba93" candidate_uuid: "92f8e028c1b1465481daa6a711279530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c76a779267cb4195aecd226588577e6a"
I20260924 05:11:38.923853 15882 raft_consensus.cc:3037] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:38.924211 16150 raft_consensus.cc:2454] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f8e028c1b1465481daa6a711279530 in term 1.
I20260924 05:11:38.924463 16218 leader_election.cc:288] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [CANDIDATE]: Term 1 election: Requested vote from peers e4e7708d96294878913f7b0ff6e6a0ee (127.0.98.3:41727), c76a779267cb4195aecd226588577e6a (127.0.98.1:35615)
I20260924 05:11:38.924609 15949 leader_election.cc:302] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f8e028c1b1465481daa6a711279530, e4e7708d96294878913f7b0ff6e6a0ee; no voters:
I20260924 05:11:38.924737 16198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.921399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35615 (local address 127.0.98.3:46793)
0924 05:11:38.921478 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.921488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.921521 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.921846 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.921846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.921848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:38.921920 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.921921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.922795 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.922796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.923837 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.923839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.924537 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.924538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.924609 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.924611 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.924640 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.924672 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":14}
I20260924 05:11:38.924770 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.921543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39667 (local address 127.0.98.2:43717)
0924 05:11:38.921581 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.921583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.921701 (+ 118us) server_negotiation.cc:407] Connection header received
0924 05:11:38.921795 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.921795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.921805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.921832 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:38.922006 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.922468 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.923795 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.924320 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.924383 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.924391 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.924612 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.924660 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:11:38.924788 15882 raft_consensus.cc:2454] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f8e028c1b1465481daa6a711279530 in term 1.
I20260924 05:11:38.924849 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.921454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.98.3:39667)
0924 05:11:38.921646 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.921656 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:38.921728 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:38.921884 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:38.921884 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:38.921886 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:38.921946 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.921947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.922542 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.922542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.923699 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:38.923702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.924449 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.924450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:38.924534 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.924538 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:38.924738 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 05:11:38.924773 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260924 05:11:38.924963 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.921589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46793 (local address 127.0.98.1:35615)
0924 05:11:38.921729 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.921731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.921737 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:11:38.921754 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.921755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.921762 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.922033 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:38.922061 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.922681 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.923926 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.924457 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.924834 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.924844 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.924863 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.924900 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260924 05:11:38.924719 16218 raft_consensus.cc:2781] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:38.925277 16016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370852666c8a46c795fad36bbee7ba93" candidate_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f8e028c1b1465481daa6a711279530" is_pre_election: true
I20260924 05:11:38.925405 16218 raft_consensus.cc:686] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 1 LEADER]: Becoming Leader. State: Replica: 92f8e028c1b1465481daa6a711279530, State: Running, Role: LEADER
I20260924 05:11:38.925495 15882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370852666c8a46c795fad36bbee7ba93" candidate_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c76a779267cb4195aecd226588577e6a" is_pre_election: true
I20260924 05:11:38.925594 15882 raft_consensus.cc:2379] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4e7708d96294878913f7b0ff6e6a0ee in current term 1: Already voted for candidate 92f8e028c1b1465481daa6a711279530 in this term.
I20260924 05:11:38.925655 16218 consensus_queue.cc:237] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:38.926270 16083 leader_election.cc:302] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4e7708d96294878913f7b0ff6e6a0ee; no voters: 92f8e028c1b1465481daa6a711279530, c76a779267cb4195aecd226588577e6a
I20260924 05:11:38.926440 16216 raft_consensus.cc:2726] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 05:11:38.926504 15737 catalog_manager.cc:5495] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92f8e028c1b1465481daa6a711279530 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "92f8e028c1b1465481daa6a711279530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } health_report { overall_health: UNKNOWN } } }
I20260924 05:11:38.937747 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.935535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.0.98.2:43717)
0924 05:11:38.935596 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:38.935598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:38.935605 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:11:38.935673 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:38.935673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:38.935687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:38.935740 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:11:38.935861 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.936304 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.937088 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:38.937262 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:38.937313 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:38.937437 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:11:38.937521 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:11:38.937578 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:38.937610 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 05:11:38.937631 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:11:38.939371 15882 raft_consensus.cc:1261] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Refusing update from remote peer 92f8e028c1b1465481daa6a711279530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:38.939424 16150 raft_consensus.cc:1261] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 FOLLOWER]: Refusing update from remote peer 92f8e028c1b1465481daa6a711279530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:38.939728 16222 consensus_queue.cc:1035] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:38.939836 16218 consensus_queue.cc:1035] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:38.941979 16225 mvcc.cc:205] Tried to move back new op lower bound from 7332768558853259264 to 7332768558800011264. Current Snapshot: MvccSnapshot[applied={T|T < 7332768558853259264}]
I20260924 05:11:38.942685 16226 mvcc.cc:205] Tried to move back new op lower bound from 7332768558853259264 to 7332768558800011264. Current Snapshot: MvccSnapshot[applied={T|T < 7332768558853259264 or (T in {7332768558853259264})}]
I20260924 05:11:38.944072 15736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:38.944183 15736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:38.944509 16227 mvcc.cc:205] Tried to move back new op lower bound from 7332768558853259264 to 7332768558800011264. Current Snapshot: MvccSnapshot[applied={T|T < 7332768558853259264 or (T in {7332768558853259264})}]
I20260924 05:11:38.944937 15736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Sending DeleteTablet for 3 replicas of tablet 370852666c8a46c795fad36bbee7ba93
I20260924 05:11:38.945277 15862 tablet_service.cc:1514] Processing DeleteTablet for tablet 370852666c8a46c795fad36bbee7ba93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36592
I20260924 05:11:38.945283 16130 tablet_service.cc:1514] Processing DeleteTablet for tablet 370852666c8a46c795fad36bbee7ba93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40818
I20260924 05:11:38.945350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 15799
I20260924 05:11:38.945345 15996 tablet_service.cc:1514] Processing DeleteTablet for tablet 370852666c8a46c795fad36bbee7ba93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52070
I20260924 05:11:38.946061 16236 tablet_replica.cc:330] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: stopping tablet replica
I20260924 05:11:38.946162 16236 raft_consensus.cc:2229] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:11:38.946203 16235 tablet_replica.cc:330] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: stopping tablet replica
I20260924 05:11:38.946250 16236 raft_consensus.cc:2258] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:38.946276 16235 raft_consensus.cc:2229] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:38.946444 16235 raft_consensus.cc:2258] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:38.946918 16235 ts_tablet_manager.cc:1905] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:38.948443 16235 ts_tablet_manager.cc:1918] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:38.948518 16235 log.cc:1196] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/370852666c8a46c795fad36bbee7ba93
I20260924 05:11:38.949012 16235 ts_tablet_manager.cc:1939] T 370852666c8a46c795fad36bbee7ba93 P e4e7708d96294878913f7b0ff6e6a0ee: Deleting consensus metadata
I20260924 05:11:38.949354 15721 catalog_manager.cc:4841] TS e4e7708d96294878913f7b0ff6e6a0ee (127.0.98.3:41727): tablet 370852666c8a46c795fad36bbee7ba93 (table test-table [id=192b76313fbc4c4dbf1593b7ea1142df]) successfully deleted
I20260924 05:11:38.950596 16236 ts_tablet_manager.cc:1905] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:11:38.951324 15722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35615 (error 108)
I20260924 05:11:38.951651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 15933
I20260924 05:11:38.952502 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.952387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35615 (local address 127.0.0.1:36598)
0924 05:11:38.952440 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.952476 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 05:11:38.952627 15722 catalog_manager.cc:4584] TS c76a779267cb4195aecd226588577e6a (127.0.98.1:35615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 370852666c8a46c795fad36bbee7ba93: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35615: connect: Connection refused (error 111)
I20260924 05:11:38.957556 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:38.957448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.0.1:52092)
0924 05:11:38.957496 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:38.957526 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:11:38.957609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 16067
I20260924 05:11:38.962052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 15705
I20260924 05:11:58.967507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--webserver_interface=127.0.98.62
--webserver_port=39649
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:11:59.050907 16238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:59.051105 16238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:59.051126 16238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:59.052565 16238 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 05:11:59.052615 16238 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 05:11:59.052629 16238 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 05:11:59.052680 16238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:11:59.052716 16238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:59.052748 16238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:59.052778 16238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:11:59.052804 16238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:11:59.054396 16238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:59.054648 16238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:59.054920 16238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:59.063182 16244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:59.063170 16245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:59.063158 16247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:59.063269 16238 server_base.cc:1034] running on GCE node
I20260924 05:11:59.063632 16238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:59.063946 16238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:59.065080 16238 hybrid_clock.cc:648] HybridClock initialized: now 1790226719065069 us; error 25 us; skew 500 ppm
Sep 24 05:11:59 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:59.080801 16238 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:11:59.081291 16238 webserver.cc:460] Webserver started at http://127.0.98.62:39649/ using document root <none> and password file <none>
I20260924 05:11:59.081532 16238 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:59.081584 16238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:59.082985 16238 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:59.083621 16254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.083808 16238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:11:59.083917 16238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "7437708936eb43faacc08fcd15e9e2fd"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "d0c7023b8d0f4b0135a55b32ad563c91"
server_key_iv: "12110b6a80c36dfc51c5fc9340bd3a03"
server_key_version: "encryptionkey@0"
I20260924 05:11:59.084303 16238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:59.090629 16238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:59.092916 16238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:59.093014 16238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:59.097304 16238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44387
I20260924 05:11:59.097354 16306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44387 every 8 connection(s)
I20260924 05:11:59.097726 16238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:11:59.099277 16307 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:11:59.099901 16307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Bootstrap starting.
I20260924 05:11:59.102321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 16238
I20260924 05:11:59.102600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35615
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44387
--webserver_port=43349
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:59.109820 16307 log.cc:824] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:59.112547 16307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:11:59.112860 16307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Bootstrap complete.
I20260924 05:11:59.115964 16307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:11:59.116291 16307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7437708936eb43faacc08fcd15e9e2fd, State: Initialized, Role: FOLLOWER
I20260924 05:11:59.116446 16307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:11:59.116510 16307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:11:59.116549 16307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:11:59.116595 16307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:11:59.117949 16307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:11:59.118057 16307 leader_election.cc:302] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7437708936eb43faacc08fcd15e9e2fd; no voters:
I20260924 05:11:59.118220 16307 leader_election.cc:288] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:11:59.118288 16311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:11:59.118451 16307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:11:59.118487 16311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 2 LEADER]: Becoming Leader. State: Replica: 7437708936eb43faacc08fcd15e9e2fd, State: Running, Role: LEADER
I20260924 05:11:59.118582 16311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:11:59.118994 16311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7437708936eb43faacc08fcd15e9e2fd. Latest consensus state: current_term: 2 leader_uuid: "7437708936eb43faacc08fcd15e9e2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } }
I20260924 05:11:59.119082 16311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:59.119599 16312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7437708936eb43faacc08fcd15e9e2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } }
I20260924 05:11:59.119871 16312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: This master's current role is: LEADER
I20260924 05:11:59.120188 16327 catalog_manager.cc:1259] Loaded cluster ID: 3734350cabd74946b9240aa877727cf2
I20260924 05:11:59.120251 16327 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: loading cluster ID for follower catalog manager: success
I20260924 05:11:59.121042 16327 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: acquiring CA information for follower catalog manager: success
I20260924 05:11:59.121388 16327 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:11:59.121505 16320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:11:59.121850 16320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:11:59.121944 16320 catalog_manager.cc:1259] Loaded cluster ID: 3734350cabd74946b9240aa877727cf2
I20260924 05:11:59.121978 16320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:11:59.122130 16320 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:11:59.122231 16320 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Loaded TSK: 0
I20260924 05:11:59.139365 16320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Generated new TSK 1
I20260924 05:11:59.139472 16320 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:11:59.202368 16309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:59.202571 16309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:59.202606 16309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:59.203713 16309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:59.203791 16309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:59.203836 16309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:11:59.205200 16309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43349
--tserver_master_addrs=127.0.98.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:59.205448 16309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:59.205679 16309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:11:59.211449 16309 server_base.cc:1034] running on GCE node
W20260924 05:11:59.211434 16335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:59.211618 16334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:59.211424 16337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:59.211895 16309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:59.212160 16309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:59.213310 16309 hybrid_clock.cc:648] HybridClock initialized: now 1790226719213290 us; error 37 us; skew 500 ppm
Sep 24 05:11:59 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:59.215751 16309 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:11:59.216223 16309 webserver.cc:460] Webserver started at http://127.0.98.1:43349/ using document root <none> and password file <none>
I20260924 05:11:59.216467 16309 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:59.216521 16309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:59.217876 16309 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:59.218549 16344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.218778 16309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:59.218894 16309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "c76a779267cb4195aecd226588577e6a"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "9735f85daf523b5edf1372a3b9bcc8c4"
server_key_iv: "b49bf7720d8227e26080381993e8e5f1"
server_key_version: "encryptionkey@0"
I20260924 05:11:59.219357 16309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:59.234215 16309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:59.236543 16309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:59.236694 16309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:59.236935 16309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:59.237416 16351 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:11:59.238828 16309 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:11:59.238885 16309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:11:59.238917 16309 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:11:59.239727 16309 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:11:59.239784 16309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:11:59.239831 16351 tablet_bootstrap.cc:492] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Bootstrap starting.
I20260924 05:11:59.246649 16309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35615
I20260924 05:11:59.246836 16458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35615 every 8 connection(s)
I20260924 05:11:59.247040 16309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:11:59.248243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 16309
I20260924 05:11:59.248533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43717
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44387
--webserver_port=42367
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:59.249948 16351 log.cc:824] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:59.252651 16351 tablet_bootstrap.cc:492] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:11:59.252950 16351 tablet_bootstrap.cc:492] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Bootstrap complete.
I20260924 05:11:59.254170 16351 ts_tablet_manager.cc:1397] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260924 05:11:59.255541 16351 raft_consensus.cc:348] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.255838 16351 raft_consensus.cc:729] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76a779267cb4195aecd226588577e6a, State: Initialized, Role: FOLLOWER
I20260924 05:11:59.255971 16351 consensus_queue.cc:260] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.256243 16351 ts_tablet_manager.cc:1428] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Sep 24 05:11:59 dist-test-slave-mr2f krb5kdc[15692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:59.263556 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.247671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:36651)
0924 05:11:59.250371 (+ 2700us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:59.250422 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:59.251558 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:59.252116 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:59.252120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:59.252497 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:59.252767 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.252774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.254606 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.254607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.256196 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.256199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.257215 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.257216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.257370 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.257966 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:59.257976 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:59.259745 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:59.261871 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:59.261877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:59.261879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:59.262137 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:59.262405 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:59.262406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:59.262406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:59.262527 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:59.262969 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:59.262975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:59.263115 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 05:11:59.263221 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 05:11:59.263365 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2651,"thread_start_us":90,"threads_started":1}
I20260924 05:11:59.264030 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.247806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36651 (local address 127.0.98.62:44387)
0924 05:11:59.250238 (+ 2432us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.250247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.251062 (+ 815us) server_negotiation.cc:407] Connection header received
0924 05:11:59.251772 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.251779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.251832 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.251942 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:59.252918 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.254471 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.256303 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.256997 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.257084 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.259884 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:59.259888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:59.259889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:59.259901 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:59.261787 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.262233 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.262236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.262236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.262335 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.262597 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.262598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.262598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.262899 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:59.262975 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.263298 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.263427 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2375,"thread_start_us":99,"threads_started":1}
I20260924 05:11:59.265129 16459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44387
I20260924 05:11:59.265214 16459 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:59.265367 16459 heartbeater.cc:511] Master 127.0.98.62:44387 requested a full tablet report, sending...
I20260924 05:11:59.266042 16271 ts_manager.cc:194] Registered new tserver with Master: c76a779267cb4195aecd226588577e6a (127.0.98.1:35615)
I20260924 05:11:59.267310 16271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36651
I20260924 05:11:59.268270 16459 heartbeater.cc:503] Master 127.0.98.62:44387 was elected leader, sending a full tablet report...
I20260924 05:11:59.274945 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.266651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35615 (local address 127.0.0.1:33172)
0924 05:11:59.270892 (+ 4241us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:59.270928 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:59.270989 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:59.271255 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:59.271258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:59.271261 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:59.271321 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.271323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.272502 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.272503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.273657 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.273660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.274589 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.274590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.274671 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.274684 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:11:59.274790 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 05:11:59.274860 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4211,"thread_start_us":95,"threads_started":1}
I20260924 05:11:59.274947 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.266943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.0.98.1:35615)
0924 05:11:59.271101 (+ 4158us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.271105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.271113 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:11:59.271141 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.271143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.271156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.271224 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:59.271406 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.272433 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.273748 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.274367 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.274506 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.274562 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.274834 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.274885 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4131,"thread_start_us":81,"threads_started":1}
I20260924 05:11:59.275519 16393 tablet_service.cc:1514] Processing DeleteTablet for tablet 370852666c8a46c795fad36bbee7ba93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33172
I20260924 05:11:59.276072 16468 tablet_replica.cc:330] stopping tablet replica
I20260924 05:11:59.276227 16468 raft_consensus.cc:2229] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:59.276310 16468 raft_consensus.cc:2258] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:59.276590 16468 ts_tablet_manager.cc:1905] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:59.278611 16468 ts_tablet_manager.cc:1918] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:59.278692 16468 log.cc:1196] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/370852666c8a46c795fad36bbee7ba93
I20260924 05:11:59.278952 16468 ts_tablet_manager.cc:1939] T 370852666c8a46c795fad36bbee7ba93 P c76a779267cb4195aecd226588577e6a: Deleting consensus metadata
I20260924 05:11:59.279445 16256 catalog_manager.cc:4841] TS c76a779267cb4195aecd226588577e6a (127.0.98.1:35615): tablet 370852666c8a46c795fad36bbee7ba93 (table test-table [id=192b76313fbc4c4dbf1593b7ea1142df]) successfully deleted
W20260924 05:11:59.333320 16463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:59.333508 16463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:59.333531 16463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:59.334602 16463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:59.334641 16463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:59.334666 16463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:11:59.336014 16463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42367
--tserver_master_addrs=127.0.98.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:59.336210 16463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:59.336364 16463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:59.340929 16476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:59.340929 16473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:59.341130 16474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:59.341104 16463 server_base.cc:1034] running on GCE node
I20260924 05:11:59.341298 16463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:59.341519 16463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:59.342672 16463 hybrid_clock.cc:648] HybridClock initialized: now 1790226719342653 us; error 35 us; skew 500 ppm
Sep 24 05:11:59 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:59.345053 16463 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:11:59.345484 16463 webserver.cc:460] Webserver started at http://127.0.98.2:42367/ using document root <none> and password file <none>
I20260924 05:11:59.345695 16463 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:59.345768 16463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:59.347092 16463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:11:59.347702 16483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.347852 16463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.347945 16463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "92f8e028c1b1465481daa6a711279530"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "dd744943439f806dc28171ff7e2df101"
server_key_iv: "6389a71752401368565218c532426d9e"
server_key_version: "encryptionkey@0"
I20260924 05:11:59.348255 16463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:59.356098 16463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:59.357729 16463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:59.357878 16463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:59.358076 16463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:59.358448 16490 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 05:11:59.359788 16490 ts_tablet_manager.cc:1852] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 05:11:59.360015 16490 ts_tablet_manager.cc:1905] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:59.362108 16490 ts_tablet_manager.cc:1918] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:59.362183 16490 log.cc:1196] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/370852666c8a46c795fad36bbee7ba93
I20260924 05:11:59.362496 16490 ts_tablet_manager.cc:1939] T 370852666c8a46c795fad36bbee7ba93 P 92f8e028c1b1465481daa6a711279530: Deleting consensus metadata
I20260924 05:11:59.362689 16463 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 05:11:59.362746 16463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 05:11:59.362839 16463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:59.362870 16463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.368789 16463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43717
I20260924 05:11:59.368845 16597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43717 every 8 connection(s)
I20260924 05:11:59.369102 16463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:11:59.374262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 16463
Sep 24 05:11:59 dist-test-slave-mr2f krb5kdc[15692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:59.374585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41727
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44387
--webserver_port=40623
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:11:59.378129 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.369729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:42191)
0924 05:11:59.369882 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:59.369921 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:59.370349 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:59.370575 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:59.370578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:59.370888 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:59.371068 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.371072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.371713 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.371714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.372516 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.372518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.372846 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.372847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.372944 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.373382 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:59.373390 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:59.375048 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:59.376769 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:59.376775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:59.376778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:59.377078 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:59.377271 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:59.377272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:59.377272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:59.377406 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:59.377670 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:59.377673 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:59.377794 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 05:11:59.377877 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:11:59.377986 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260924 05:11:59.378165 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.369770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42191 (local address 127.0.98.62:44387)
0924 05:11:59.369817 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.369819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.369996 (+ 177us) server_negotiation.cc:407] Connection header received
0924 05:11:59.370436 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.370437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.370448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.370504 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:59.371147 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.371638 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.372589 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.372779 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.372833 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.375139 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:59.375141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:59.375141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:59.375148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:59.376688 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.377161 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.377161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.377162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.377209 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.377470 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.377470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.377471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.377607 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:59.377675 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.377924 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.378022 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:11:59.378748 16598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44387
I20260924 05:11:59.378873 16598 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:59.379040 16598 heartbeater.cc:511] Master 127.0.98.62:44387 requested a full tablet report, sending...
I20260924 05:11:59.379436 16271 ts_manager.cc:194] Registered new tserver with Master: 92f8e028c1b1465481daa6a711279530 (127.0.98.2:43717)
I20260924 05:11:59.379998 16271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42191
W20260924 05:11:59.454341 16601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:11:59.454537 16601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:11:59.454559 16601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:11:59.455662 16601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:11:59.455698 16601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:11:59.455724 16601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:11:59.457068 16601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40623
--tserver_master_addrs=127.0.98.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:11:59.457233 16601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:11:59.457412 16601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:11:59.462206 16609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:11:59.462214 16607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:59.462231 16601 server_base.cc:1034] running on GCE node
W20260924 05:11:59.462216 16606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:11:59.462651 16601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:11:59.462914 16601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:11:59.464066 16601 hybrid_clock.cc:648] HybridClock initialized: now 1790226719464058 us; error 39 us; skew 500 ppm
Sep 24 05:11:59 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:11:59.466384 16601 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:11:59.466729 16601 webserver.cc:460] Webserver started at http://127.0.98.3:40623/ using document root <none> and password file <none>
I20260924 05:11:59.466943 16601 fs_manager.cc:359] Metadata directory not provided
I20260924 05:11:59.466995 16601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:11:59.468174 16601 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:59.468699 16616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.468843 16601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.468928 16601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "e4e7708d96294878913f7b0ff6e6a0ee"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "e919127dbbbcfa06cc03f7f0a5c3ccb8"
server_key_iv: "fa9c0b96a6c3106edcbbac950624542a"
server_key_version: "encryptionkey@0"
I20260924 05:11:59.469249 16601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:11:59.483456 16601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:11:59.485033 16601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:11:59.485131 16601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:11:59.485301 16601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:11:59.485520 16601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:11:59.485548 16601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.485566 16601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:11:59.485579 16601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.491217 16601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41727
I20260924 05:11:59.491267 16729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41727 every 8 connection(s)
I20260924 05:11:59.491556 16601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:11:59 dist-test-slave-mr2f krb5kdc[15692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:11:59.499531 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.492172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40367 (local address 127.0.98.62:44387)
0924 05:11:59.492214 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.492216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.492434 (+ 218us) server_negotiation.cc:407] Connection header received
0924 05:11:59.492907 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.492908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.492919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.492976 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:59.493767 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.494215 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.495110 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.495290 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.495373 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.497194 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:59.497195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:59.497195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:59.497201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:59.498375 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.498720 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.498720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.498721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.498823 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.499026 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.499026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.499026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.499126 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:59.499193 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.499411 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.499470 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:11:59.499693 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.492168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:40367)
0924 05:11:59.492325 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:59.492355 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:59.492802 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:59.493035 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:59.493039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:59.493519 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:11:59.493673 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.493677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.494299 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.494300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.495025 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.495027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.495380 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.495381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.495493 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.495849 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:11:59.495855 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:11:59.497102 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:11:59.498454 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:59.498459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:59.498461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:59.498659 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:59.498880 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:11:59.498881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:11:59.498881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:11:59.498971 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:11:59.499196 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:11:59.499200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:11:59.499297 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 05:11:59.499383 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 05:11:59.499488 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260924 05:11:59.500177 16730 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44387
I20260924 05:11:59.500308 16730 heartbeater.cc:464] Registering TS with master...
I20260924 05:11:59.500383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 16601
I20260924 05:11:59.500492 16730 heartbeater.cc:511] Master 127.0.98.62:44387 requested a full tablet report, sending...
I20260924 05:11:59.500872 16271 ts_manager.cc:194] Registered new tserver with Master: e4e7708d96294878913f7b0ff6e6a0ee (127.0.98.3:41727)
I20260924 05:11:59.501418 16271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40367
I20260924 05:11:59.506330 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.500784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.0.98.62:44387)
0924 05:11:59.500823 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.500825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.500984 (+ 159us) server_negotiation.cc:407] Connection header received
0924 05:11:59.501066 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.501067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.501075 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.501139 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:59.501862 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.502421 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.503205 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.503425 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.503483 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.504433 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:59.504434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:59.504434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:59.504439 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:59.505517 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.505808 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.505808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.505808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.505851 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.505960 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.505960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.505961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.506034 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:59.506133 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.506200 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.506266 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:11:59.507040 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:11:59.509225 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.507376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58352 (local address 127.0.98.62:44387)
0924 05:11:59.507432 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.507434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.507482 (+ 48us) server_negotiation.cc:407] Connection header received
0924 05:11:59.507537 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.507538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.507563 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.507622 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:11:59.507683 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.508091 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.508748 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.508911 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.508958 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.509069 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:11:59.509103 (+ 34us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 05:11:59.509190 (+ 87us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58352: token expired
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 05:11:59.509289 16462 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58352: token expired
W20260924 05:11:59.509366 16200 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:44387)
I20260924 05:11:59.514715 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.509571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.0.98.62:44387)
0924 05:11:59.509618 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.509620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.509673 (+ 53us) server_negotiation.cc:407] Connection header received
0924 05:11:59.509742 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.509743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.509751 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.509801 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:11:59.510257 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.510686 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.511401 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.511579 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.511660 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.512625 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:11:59.512626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:11:59.512626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:11:59.512632 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:11:59.513656 (+ 1024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.514002 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.514003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.514004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.514057 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:11:59.514241 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:11:59.514242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:11:59.514242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:11:59.514360 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:11:59.514467 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.514564 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.514630 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:11:59.515687 16271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:11:59.517225 16271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:11:59.519192 16393 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba34a00c68843f18a739fee1fef0283 (DEFAULT_TABLE table=test-table [id=c04020bf6c3a401db8dea2d5b58425f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:59.519371 16393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba34a00c68843f18a739fee1fef0283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:59.520833 16351 tablet_bootstrap.cc:492] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: Bootstrap starting.
I20260924 05:11:59.521199 16351 tablet_bootstrap.cc:654] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:59.521730 16351 tablet_bootstrap.cc:492] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: No bootstrap required, opened a new log
I20260924 05:11:59.521773 16351 ts_tablet_manager.cc:1397] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:11:59.521883 16351 raft_consensus.cc:348] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.521943 16351 raft_consensus.cc:374] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:59.521982 16351 raft_consensus.cc:729] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76a779267cb4195aecd226588577e6a, State: Initialized, Role: FOLLOWER
I20260924 05:11:59.522020 16351 consensus_queue.cc:260] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.522104 16351 ts_tablet_manager.cc:1428] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:11:59.522650 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.518931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41727 (local address 127.0.0.1:33812)
0924 05:11:59.519046 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:59.519057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:59.519111 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:59.519442 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:59.519444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:59.519446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:59.519484 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.519485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.520452 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.520453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.521471 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.521472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.522453 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.522454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.522510 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.522514 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:59.522553 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 05:11:59.522586 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260924 05:11:59.522753 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.519093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.0.98.3:41727)
0924 05:11:59.519287 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.519291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.519301 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:11:59.519336 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.519338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.519352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.519426 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:59.519545 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.520378 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.521542 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.522375 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.522453 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.522607 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.522633 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.522671 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260924 05:11:59.523906 16664 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba34a00c68843f18a739fee1fef0283 (DEFAULT_TABLE table=test-table [id=c04020bf6c3a401db8dea2d5b58425f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:59.524183 16664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba34a00c68843f18a739fee1fef0283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:59.524250 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.518843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.0.1:53312)
0924 05:11:59.518935 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:59.518946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:59.518980 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:59.521173 (+ 2193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:59.521174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:59.521175 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:59.521204 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.521205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.522392 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.522393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.523293 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.523296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.524055 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.524056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.524109 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.524112 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:11:59.524155 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:11:59.524192 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 05:11:59.524276 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.518939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.0.98.2:43717)
0924 05:11:59.521009 (+ 2070us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.521013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.521022 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:11:59.521059 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.521061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.521074 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.521258 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:59.521329 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.522301 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.523379 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.523949 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.524083 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.524143 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.524168 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.524201 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2018,"thread_start_us":105,"threads_started":1}
I20260924 05:11:59.524708 16532 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba34a00c68843f18a739fee1fef0283 (DEFAULT_TABLE table=test-table [id=c04020bf6c3a401db8dea2d5b58425f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:11:59.524962 16532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba34a00c68843f18a739fee1fef0283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:11:59.526427 16490 tablet_bootstrap.cc:492] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: Bootstrap starting.
I20260924 05:11:59.526850 16739 tablet_bootstrap.cc:492] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: Bootstrap starting.
I20260924 05:11:59.527354 16490 tablet_bootstrap.cc:654] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:59.527510 16739 tablet_bootstrap.cc:654] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: Neither blocks nor log segments found. Creating new log.
I20260924 05:11:59.527676 16490 log.cc:824] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:59.528051 16739 log.cc:824] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: Log is configured to *not* fsync() on all Append() calls
I20260924 05:11:59.528394 16490 tablet_bootstrap.cc:492] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: No bootstrap required, opened a new log
I20260924 05:11:59.528460 16490 ts_tablet_manager.cc:1397] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:11:59.528740 16739 tablet_bootstrap.cc:492] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: No bootstrap required, opened a new log
I20260924 05:11:59.528802 16739 ts_tablet_manager.cc:1397] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:11:59.530367 16490 raft_consensus.cc:348] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.530531 16490 raft_consensus.cc:374] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:59.530565 16490 raft_consensus.cc:729] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f8e028c1b1465481daa6a711279530, State: Initialized, Role: FOLLOWER
I20260924 05:11:59.530625 16739 raft_consensus.cc:348] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.530717 16490 consensus_queue.cc:260] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.530798 16739 raft_consensus.cc:374] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:11:59.530843 16739 raft_consensus.cc:729] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e7708d96294878913f7b0ff6e6a0ee, State: Initialized, Role: FOLLOWER
I20260924 05:11:59.530963 16739 consensus_queue.cc:260] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.531026 16598 heartbeater.cc:503] Master 127.0.98.62:44387 was elected leader, sending a full tablet report...
I20260924 05:11:59.531217 16739 ts_tablet_manager.cc:1428] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 05:11:59.531234 16730 heartbeater.cc:503] Master 127.0.98.62:44387 was elected leader, sending a full tablet report...
I20260924 05:11:59.531380 16490 ts_tablet_manager.cc:1428] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 05:11:59.532117 16743 raft_consensus.cc:482] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:11:59.532217 16743 raft_consensus.cc:504] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.532482 16743 leader_election.cc:288] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f8e028c1b1465481daa6a711279530 (127.0.98.2:43717), c76a779267cb4195aecd226588577e6a (127.0.98.1:35615)
I20260924 05:11:59.535173 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.532603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43717 (local address 127.0.98.3:60679)
0924 05:11:59.532638 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:59.532650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:59.532693 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:59.532856 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:59.532857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:59.532858 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:59.532911 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.532913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.533523 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.533523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.534308 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.534310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.534933 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.534934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.535014 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.535018 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:59.535067 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:11:59.535100 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 05:11:59.535197 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.532701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60679 (local address 127.0.98.2:43717)
0924 05:11:59.532741 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.532744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.532752 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:11:59.532774 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.532775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.532785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.532846 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:59.532973 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.533454 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.534397 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.534867 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.534921 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.534928 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.535091 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.535125 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:11:59.535313 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.532694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52633 (local address 127.0.98.1:35615)
0924 05:11:59.532747 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.532749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.532809 (+ 60us) server_negotiation.cc:407] Connection header received
0924 05:11:59.532865 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.532866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.532882 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.532944 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:11:59.533113 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.533589 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.534507 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.534961 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.535033 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.535042 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.535194 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.535238 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:11:59.535431 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.532603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35615 (local address 127.0.98.3:52633)
0924 05:11:59.532754 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:59.532764 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:11:59.532818 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:11:59.532963 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:11:59.532963 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:11:59.532965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:11:59.533026 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.533027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.533671 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.533672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.534425 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:11:59.534427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.535038 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.535039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:11:59.535121 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.535125 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:11:59.535179 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 05:11:59.535216 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":14,"thread_start_us":66,"threads_started":1}
I20260924 05:11:59.535444 16552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba34a00c68843f18a739fee1fef0283" candidate_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f8e028c1b1465481daa6a711279530" is_pre_election: true
I20260924 05:11:59.535562 16552 raft_consensus.cc:2454] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e7708d96294878913f7b0ff6e6a0ee in term 0.
I20260924 05:11:59.535674 16413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba34a00c68843f18a739fee1fef0283" candidate_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c76a779267cb4195aecd226588577e6a" is_pre_election: true
I20260924 05:11:59.535749 16617 leader_election.cc:302] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f8e028c1b1465481daa6a711279530, e4e7708d96294878913f7b0ff6e6a0ee; no voters:
I20260924 05:11:59.535816 16413 raft_consensus.cc:2454] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e7708d96294878913f7b0ff6e6a0ee in term 0.
I20260924 05:11:59.535892 16743 raft_consensus.cc:2781] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:11:59.535950 16743 raft_consensus.cc:482] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:11:59.535972 16743 raft_consensus.cc:3037] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:59.536553 16743 raft_consensus.cc:504] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.536679 16743 leader_election.cc:288] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [CANDIDATE]: Term 1 election: Requested vote from peers 92f8e028c1b1465481daa6a711279530 (127.0.98.2:43717), c76a779267cb4195aecd226588577e6a (127.0.98.1:35615)
I20260924 05:11:59.536788 16413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba34a00c68843f18a739fee1fef0283" candidate_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c76a779267cb4195aecd226588577e6a"
I20260924 05:11:59.536777 16552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba34a00c68843f18a739fee1fef0283" candidate_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f8e028c1b1465481daa6a711279530"
I20260924 05:11:59.536857 16552 raft_consensus.cc:3037] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:59.536859 16413 raft_consensus.cc:3037] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:11:59.537433 16552 raft_consensus.cc:2454] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e7708d96294878913f7b0ff6e6a0ee in term 1.
I20260924 05:11:59.537597 16617 leader_election.cc:302] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f8e028c1b1465481daa6a711279530, e4e7708d96294878913f7b0ff6e6a0ee; no voters:
I20260924 05:11:59.537786 16743 raft_consensus.cc:2781] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:11:59.537907 16413 raft_consensus.cc:2454] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e7708d96294878913f7b0ff6e6a0ee in term 1.
I20260924 05:11:59.537945 16743 raft_consensus.cc:686] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 LEADER]: Becoming Leader. State: Replica: e4e7708d96294878913f7b0ff6e6a0ee, State: Running, Role: LEADER
I20260924 05:11:59.538095 16743 consensus_queue.cc:237] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } }
I20260924 05:11:59.538740 16271 catalog_manager.cc:5495] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee reported cstate change: term changed from 0 to 1, leader changed from <none> to e4e7708d96294878913f7b0ff6e6a0ee (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4e7708d96294878913f7b0ff6e6a0ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 } health_report { overall_health: UNKNOWN } } }
I20260924 05:11:59.546398 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.544553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.0.98.3:41727)
0924 05:11:59.544608 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:11:59.544610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:11:59.544617 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:11:59.544643 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:11:59.544644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:11:59.544667 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:11:59.544719 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:11:59.544792 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.545199 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.545848 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:11:59.546015 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:11:59.546063 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:11:59.546176 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:11:59.546251 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:11:59.546295 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:11:59.546325 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 05:11:59.546338 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:11:59.548004 16552 raft_consensus.cc:1261] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [term 1 FOLLOWER]: Refusing update from remote peer e4e7708d96294878913f7b0ff6e6a0ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:59.548004 16413 raft_consensus.cc:1261] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Refusing update from remote peer e4e7708d96294878913f7b0ff6e6a0ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:11:59.548346 16745 consensus_queue.cc:1035] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "c76a779267cb4195aecd226588577e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:59.548450 16743 consensus_queue.cc:1035] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f8e028c1b1465481daa6a711279530" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:11:59.550346 16749 mvcc.cc:205] Tried to move back new op lower bound from 7332768643266297856 to 7332768643228700672. Current Snapshot: MvccSnapshot[applied={T|T < 7332768643266297856}]
I20260924 05:11:59.552568 16271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:59.552649 16271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:11:59.553278 16271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Sending DeleteTablet for 3 replicas of tablet 1ba34a00c68843f18a739fee1fef0283
I20260924 05:11:59.553563 16532 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba34a00c68843f18a739fee1fef0283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53312
I20260924 05:11:59.553577 16664 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba34a00c68843f18a739fee1fef0283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33812
I20260924 05:11:59.553766 16757 tablet_replica.cc:330] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: stopping tablet replica
I20260924 05:11:59.553757 16393 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba34a00c68843f18a739fee1fef0283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:11:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33172
I20260924 05:11:59.553846 16468 tablet_replica.cc:330] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: stopping tablet replica
I20260924 05:11:59.553855 16757 raft_consensus.cc:2229] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:59.553896 16468 raft_consensus.cc:2229] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:11:59.553947 16468 raft_consensus.cc:2258] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:59.553966 16757 raft_consensus.cc:2258] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:59.554265 16468 ts_tablet_manager.cc:1905] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:59.554345 16757 ts_tablet_manager.cc:1905] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:11:59.554879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 16309
I20260924 05:11:59.560073 16758 tablet_replica.cc:330] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: stopping tablet replica
I20260924 05:11:59.560179 16758 raft_consensus.cc:2229] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:11:59.560302 16758 raft_consensus.cc:2258] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:11:59.560287 16757 ts_tablet_manager.cc:1918] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:59.560411 16757 log.cc:1196] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/1ba34a00c68843f18a739fee1fef0283
I20260924 05:11:59.560734 16757 ts_tablet_manager.cc:1939] T 1ba34a00c68843f18a739fee1fef0283 P 92f8e028c1b1465481daa6a711279530: Deleting consensus metadata
I20260924 05:11:59.560750 16758 ts_tablet_manager.cc:1905] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:11:59.560837 16256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35615 (error 108)
I20260924 05:11:59.561257 16255 catalog_manager.cc:4841] TS 92f8e028c1b1465481daa6a711279530 (127.0.98.2:43717): tablet 1ba34a00c68843f18a739fee1fef0283 (table test-table [id=c04020bf6c3a401db8dea2d5b58425f0]) successfully deleted
I20260924 05:11:59.561677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 16463
I20260924 05:11:59.561889 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:11:59.561765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35615 (local address 127.0.0.1:33188)
0924 05:11:59.561810 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:11:59.561842 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:11:59.562597 16758 ts_tablet_manager.cc:1918] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:11:59.562656 16758 log.cc:1196] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/1ba34a00c68843f18a739fee1fef0283
W20260924 05:11:59.562527 16256 catalog_manager.cc:4584] TS c76a779267cb4195aecd226588577e6a (127.0.98.1:35615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ba34a00c68843f18a739fee1fef0283: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35615: connect: Connection refused (error 111)
I20260924 05:11:59.562949 16758 ts_tablet_manager.cc:1939] T 1ba34a00c68843f18a739fee1fef0283 P e4e7708d96294878913f7b0ff6e6a0ee: Deleting consensus metadata
I20260924 05:11:59.563301 16255 catalog_manager.cc:4841] TS e4e7708d96294878913f7b0ff6e6a0ee (127.0.98.3:41727): tablet 1ba34a00c68843f18a739fee1fef0283 (table test-table [id=c04020bf6c3a401db8dea2d5b58425f0]) successfully deleted
I20260924 05:11:59.567371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 16601
I20260924 05:11:59.571425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 16238
I20260924 05:12:19.576670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--webserver_interface=127.0.98.62
--webserver_port=39649
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:19.654700 16760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:19.654953 16760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:19.654987 16760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:19.656121 16760 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 05:12:19.656189 16760 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 05:12:19.656213 16760 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 05:12:19.656241 16760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:19.656283 16760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:19.656316 16760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:19.656345 16760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:19.656373 16760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:19.657776 16760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:19.658033 16760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:19.658244 16760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:19.663338 16765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:19.663501 16768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:19.663369 16760 server_base.cc:1034] running on GCE node
W20260924 05:12:19.663398 16766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:19.663914 16760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:19.664171 16760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:19.665308 16760 hybrid_clock.cc:648] HybridClock initialized: now 1790226739665300 us; error 31 us; skew 500 ppm
Sep 24 05:12:19 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:19.676291 16760 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:12:19.676699 16760 webserver.cc:460] Webserver started at http://127.0.98.62:39649/ using document root <none> and password file <none>
I20260924 05:12:19.676911 16760 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:19.676976 16760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:19.678224 16760 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:12:19.678885 16775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:19.679052 16760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:19.679164 16760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "7437708936eb43faacc08fcd15e9e2fd"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "d0c7023b8d0f4b0135a55b32ad563c91"
server_key_iv: "12110b6a80c36dfc51c5fc9340bd3a03"
server_key_version: "encryptionkey@0"
I20260924 05:12:19.679479 16760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:19.694367 16760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:19.696507 16760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:19.696660 16760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:19.700644 16760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44387
I20260924 05:12:19.700704 16827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44387 every 8 connection(s)
I20260924 05:12:19.701071 16760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:19.701520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 16760
I20260924 05:12:19.701809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35615
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44387
--webserver_port=43349
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:12:19.702525 16828 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:12:19.703378 16828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Bootstrap starting.
I20260924 05:12:19.711434 16828 log.cc:824] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:19.713483 16828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:12:19.713780 16828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Bootstrap complete.
I20260924 05:12:19.715970 16828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:12:19.716174 16828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7437708936eb43faacc08fcd15e9e2fd, State: Initialized, Role: FOLLOWER
I20260924 05:12:19.716334 16828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:12:19.716395 16828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:19.716442 16828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:19.716490 16828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:12:19.717478 16828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:12:19.717587 16828 leader_election.cc:302] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7437708936eb43faacc08fcd15e9e2fd; no voters:
I20260924 05:12:19.717787 16828 leader_election.cc:288] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 05:12:19.717825 16832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 05:12:19.717952 16832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [term 3 LEADER]: Becoming Leader. State: Replica: 7437708936eb43faacc08fcd15e9e2fd, State: Running, Role: LEADER
I20260924 05:12:19.718037 16828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:19.718037 16832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } }
I20260924 05:12:19.718444 16834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7437708936eb43faacc08fcd15e9e2fd. Latest consensus state: current_term: 3 leader_uuid: "7437708936eb43faacc08fcd15e9e2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } }
I20260924 05:12:19.718688 16834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:19.718446 16833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7437708936eb43faacc08fcd15e9e2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7437708936eb43faacc08fcd15e9e2fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } }
I20260924 05:12:19.718958 16842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:19.718973 16833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:19.719740 16842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:19.719841 16842 catalog_manager.cc:1259] Loaded cluster ID: 3734350cabd74946b9240aa877727cf2
I20260924 05:12:19.719873 16842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:19.721202 16842 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:19.721369 16842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Loaded TSK: 0
I20260924 05:12:19.721416 16842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7437708936eb43faacc08fcd15e9e2fd: Loaded TSK: 1
F20260924 05:12:19.721644 16842 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6354d6f9ad google::LogMessage::Fail() at ??:0
@ 0x7f6354d7288a google::LogMessage::SendToLog() at ??:0
@ 0x7f6354d6f4de google::LogMessage::Flush() at ??:0
@ 0x7f6354d70a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f63574dc104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f63574d9d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f635ccc91cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f635ccc948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f635ccc9594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f635cc8fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f635cc99ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6356c3d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6356c32954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6357a966db start_thread at ??:0
@ 0x7f635386d61f clone at ??:0
W20260924 05:12:19.798137 16830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:19.798322 16830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:19.798345 16830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:19.799556 16830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:19.799598 16830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:19.799631 16830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:12:19.801052 16830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43349
--tserver_master_addrs=127.0.98.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:19.801293 16830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:19.801537 16830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:19.806619 16869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:19.807305 16830 server_base.cc:1034] running on GCE node
W20260924 05:12:19.807469 16870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:19.807471 16872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:19.807689 16830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:19.807915 16830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:19.809065 16830 hybrid_clock.cc:648] HybridClock initialized: now 1790226739809039 us; error 45 us; skew 500 ppm
Sep 24 05:12:19 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:19.811703 16830 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:12:19.812180 16830 webserver.cc:460] Webserver started at http://127.0.98.1:43349/ using document root <none> and password file <none>
I20260924 05:12:19.812419 16830 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:19.812484 16830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:19.813721 16830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:12:19.814509 16879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:19.814668 16830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:12:19.814751 16830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "c76a779267cb4195aecd226588577e6a"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "9735f85daf523b5edf1372a3b9bcc8c4"
server_key_iv: "b49bf7720d8227e26080381993e8e5f1"
server_key_version: "encryptionkey@0"
I20260924 05:12:19.815101 16830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:19.849568 16830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:19.851976 16830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:19.852140 16830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:19.852392 16830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:19.852842 16886 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 05:12:19.854704 16886 ts_tablet_manager.cc:1852] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 05:12:19.855125 16886 ts_tablet_manager.cc:1905] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:19.857851 16886 ts_tablet_manager.cc:1918] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:19.857949 16886 log.cc:1196] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/wal/wals/1ba34a00c68843f18a739fee1fef0283
I20260924 05:12:19.858254 16886 ts_tablet_manager.cc:1939] T 1ba34a00c68843f18a739fee1fef0283 P c76a779267cb4195aecd226588577e6a: Deleting consensus metadata
I20260924 05:12:19.858472 16830 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 05:12:19.858536 16830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 05:12:19.858580 16830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:19.858608 16830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:19.865980 16830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35615
I20260924 05:12:19.866068 16993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35615 every 8 connection(s)
I20260924 05:12:19.866341 16830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:12:19.867180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 16830
I20260924 05:12:19.867492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43717
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44387
--webserver_port=42367
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:19.905602 16996 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:12:19.867084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:54381)
0924 05:12:19.867291 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:19.867339 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:19.867993 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:19.905391 (+ 37398us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
W20260924 05:12:19.905876 16994 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44387 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44387: Client connection negotiation failed: client connection to 127.0.98.62:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:12:19.906278 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:19.906148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:57147)
0924 05:12:19.906199 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:19.906255 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 05:12:19.906622 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:19.906537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:42257)
0924 05:12:19.906582 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:19.906606 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 05:12:19.906720 16994 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:12:19.951090 16997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:19.951273 16997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:19.951295 16997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:19.952345 16997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:19.952376 16997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:19.952400 16997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:12:19.953662 16997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42367
--tserver_master_addrs=127.0.98.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:19.953852 16997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:19.954012 16997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:12:19.958997 16997 server_base.cc:1034] running on GCE node
W20260924 05:12:19.958961 17003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:19.958969 17002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:19.958969 17005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:19.959429 16997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:19.959669 16997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:19.960829 16997 hybrid_clock.cc:648] HybridClock initialized: now 1790226739960828 us; error 36 us; skew 500 ppm
Sep 24 05:12:19 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:19.963390 16997 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:12:19.963809 16997 webserver.cc:460] Webserver started at http://127.0.98.2:42367/ using document root <none> and password file <none>
I20260924 05:12:19.964040 16997 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:19.964110 16997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:19.965410 16997 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 05:12:19.965975 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:19.966144 16997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:19.966245 16997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "92f8e028c1b1465481daa6a711279530"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "dd744943439f806dc28171ff7e2df101"
server_key_iv: "6389a71752401368565218c532426d9e"
server_key_version: "encryptionkey@0"
I20260924 05:12:19.966562 16997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:19.984361 16997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:19.986073 16997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:19.986236 16997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:19.986457 16997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:19.986756 16997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:19.986865 16997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:19.986907 16997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:19.986933 16997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:19.992703 16997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43717
I20260924 05:12:19.992762 17125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43717 every 8 connection(s)
I20260924 05:12:19.993106 16997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:12:19.994141 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:19.993803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:48979)
0924 05:12:19.993946 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:19.994009 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
W20260924 05:12:19.994531 17126 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44387 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44387: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
I20260924 05:12:19.994822 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:19.994694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:33307)
0924 05:12:19.994741 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:19.994799 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 05:12:19.995144 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:19.995068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:43169)
0924 05:12:19.995109 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:19.995131 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 05:12:19.995265 17126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:12:20.002982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 16997
I20260924 05:12:20.003252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41727
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44387
--webserver_port=40623
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:20.081013 17129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:20.081248 17129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:20.081285 17129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:20.082409 17129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:20.082474 17129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:20.082518 17129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:12:20.083961 17129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40623
--tserver_master_addrs=127.0.98.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:20.084216 17129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:20.084448 17129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:20.089000 17137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:20.089025 17135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:20.089161 17129 server_base.cc:1034] running on GCE node
W20260924 05:12:20.089303 17134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:20.089532 17129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:20.089793 17129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:20.090953 17129 hybrid_clock.cc:648] HybridClock initialized: now 1790226740090940 us; error 42 us; skew 500 ppm
Sep 24 05:12:20 dist-test-slave-mr2f krb5kdc[15692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:20.093376 17129 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:12:20.093765 17129 webserver.cc:460] Webserver started at http://127.0.98.3:40623/ using document root <none> and password file <none>
I20260924 05:12:20.093987 17129 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:20.094048 17129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:20.095355 17129 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:12:20.095929 17144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:20.096113 17129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:20.096202 17129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "e4e7708d96294878913f7b0ff6e6a0ee"
format_stamp: "Formatted at 2026-09-24 05:11:38 on dist-test-slave-mr2f"
server_key: "e919127dbbbcfa06cc03f7f0a5c3ccb8"
server_key_iv: "fa9c0b96a6c3106edcbbac950624542a"
server_key_version: "encryptionkey@0"
I20260924 05:12:20.096494 17129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:20.104529 17129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:20.106168 17129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:20.106315 17129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:20.106518 17129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:20.106858 17129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:20.106910 17129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:20.106950 17129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:20.106977 17129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:20.112617 17129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41727
I20260924 05:12:20.112661 17257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41727 every 8 connection(s)
I20260924 05:12:20.112967 17129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:12:20.113890 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:20.113560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:50033)
0924 05:12:20.113688 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:20.113747 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
W20260924 05:12:20.114265 17258 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44387 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44387: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
I20260924 05:12:20.114539 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:20.114455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:36581)
0924 05:12:20.114498 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:20.114523 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:12:20.114830 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:20.114739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:53271)
0924 05:12:20.114791 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:20.114811 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260924 05:12:20.114944 17258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:12:20.117710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 17129
I20260924 05:12:20.907315 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:20.907090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:55297)
0924 05:12:20.907221 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:20.907256 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260924 05:12:20.995728 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:20.995542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:52877)
0924 05:12:20.995648 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:20.995683 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260924 05:12:21.115428 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:21.115241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:34641)
0924 05:12:21.115358 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:21.115387 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260924 05:12:21.907997 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:21.907776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:33159)
0924 05:12:21.907913 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:21.907950 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260924 05:12:21.996340 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:21.996141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:36185)
0924 05:12:21.996269 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:21.996301 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":55,"threads_started":1}
I20260924 05:12:22.115988 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:22.115806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:56537)
0924 05:12:22.115924 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:22.115958 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":53,"threads_started":1}
I20260924 05:12:22.908758 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:22.908439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:45277)
0924 05:12:22.908660 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:22.908695 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":56,"threads_started":1}
I20260924 05:12:22.996985 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:22.996788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:58623)
0924 05:12:22.996916 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:22.996952 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260924 05:12:23.116559 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:23.116369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:45249)
0924 05:12:23.116489 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:23.116523 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260924 05:12:23.909530 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:23.909340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:59267)
0924 05:12:23.909455 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:23.909486 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260924 05:12:23.997574 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:23.997400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:47377)
0924 05:12:23.997513 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:23.997542 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260924 05:12:24.117180 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:24.116960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:43951)
0924 05:12:24.117111 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:24.117146 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260924 05:12:24.910261 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:24.910041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:51731)
0924 05:12:24.910180 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:24.910216 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
I20260924 05:12:24.998165 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:24.997991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:56663)
0924 05:12:24.998099 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:24.998131 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260924 05:12:25.117756 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:25.117555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:39481)
0924 05:12:25.117680 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:25.117710 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260924 05:12:25.910920 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:25.910682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:57063)
0924 05:12:25.910840 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:25.910874 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":56,"threads_started":1}
I20260924 05:12:25.999059 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:25.998867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:35397)
0924 05:12:25.998992 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:25.999025 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260924 05:12:26.118331 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:26.118148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:53073)
0924 05:12:26.118259 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:26.118297 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260924 05:12:26.911595 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:26.911381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:60185)
0924 05:12:26.911505 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:26.911549 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260924 05:12:26.999673 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:26.999442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:49149)
0924 05:12:26.999603 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:26.999637 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260924 05:12:27.119069 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:27.118856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:41557)
0924 05:12:27.118994 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:27.119027 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260924 05:12:27.912241 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:27.912020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:39541)
0924 05:12:27.912141 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:27.912172 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260924 05:12:28.000313 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:28.000108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:45007)
0924 05:12:28.000234 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:28.000270 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260924 05:12:28.119661 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:28.119477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:41103)
0924 05:12:28.119591 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:28.119621 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260924 05:12:28.912909 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:28.912708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:40563)
0924 05:12:28.912825 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:28.912863 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260924 05:12:29.000947 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:29.000733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:39239)
0924 05:12:29.000882 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:29.000912 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260924 05:12:29.120287 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:29.120088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:56175)
0924 05:12:29.120216 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:29.120250 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260924 05:12:29.913580 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:29.913387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:35345)
0924 05:12:29.913504 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:29.913534 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260924 05:12:30.001636 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:30.001412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:34173)
0924 05:12:30.001561 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:30.001591 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260924 05:12:30.120870 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:30.120680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:46191)
0924 05:12:30.120803 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:30.120832 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260924 05:12:30.914325 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:30.914108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:35159)
0924 05:12:30.914242 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:30.914279 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260924 05:12:31.002239 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:31.002028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:35187)
0924 05:12:31.002168 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:31.002204 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260924 05:12:31.121450 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:31.121255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:46993)
0924 05:12:31.121368 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:31.121404 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260924 05:12:31.915019 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:31.914818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:60347)
0924 05:12:31.914935 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:31.914966 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260924 05:12:32.002912 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:32.002683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:42575)
0924 05:12:32.002838 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:32.002871 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260924 05:12:32.122088 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:32.121886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:49049)
0924 05:12:32.122020 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:32.122053 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260924 05:12:32.915681 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:32.915467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:54929)
0924 05:12:32.915598 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:32.915634 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260924 05:12:33.003518 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:33.003325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:46247)
0924 05:12:33.003445 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:33.003478 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260924 05:12:33.122692 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:33.122504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:48291)
0924 05:12:33.122617 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:33.122651 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260924 05:12:33.916354 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:33.916159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:44589)
0924 05:12:33.916273 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:33.916308 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260924 05:12:34.004108 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:34.003923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:55305)
0924 05:12:34.004037 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:34.004066 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260924 05:12:34.123278 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:34.123098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:40607)
0924 05:12:34.123205 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:34.123239 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260924 05:12:34.917058 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:34.916831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.1:41231)
0924 05:12:34.916972 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:34.917016 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260924 05:12:35.004760 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:35.004577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.2:53159)
0924 05:12:35.004685 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:35.004713 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260924 05:12:35.123944 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:35.123754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.98.3:52883)
0924 05:12:35.123876 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:35.123906 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260924 05:12:35.128314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 16830
I20260924 05:12:35.132485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 16997
I20260924 05:12:35.136564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 17129
2026-09-24T05:12:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58083 ms)
[----------] 30 tests from SecurityITest (227401 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:35 dist-test-slave-mr2f krb5kdc[17309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:35 dist-test-slave-mr2f krb5kdc[17309](info): set up 2 sockets
Sep 24 05:12:35 dist-test-slave-mr2f krb5kdc[17309](info): commencing operation
krb5kdc: starting...
W20260924 05:12:36.162304 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:36 dist-test-slave-mr2f krb5kdc[17309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:12:36.277629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:36.355373 17322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:36.355639 17322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:36.355676 17322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:36.356829 17322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:36.356905 17322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:36.356928 17322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:36.356950 17322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:36.356977 17322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:36.358439 17322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:36.358709 17322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:36.358950 17322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:12:36.363976 17322 server_base.cc:1034] running on GCE node
W20260924 05:12:36.363948 17328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:36.363957 17327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:36.363957 17330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:36.364413 17322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:36.364706 17322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:36.365850 17322 hybrid_clock.cc:648] HybridClock initialized: now 1790226756365824 us; error 43 us; skew 500 ppm
Sep 24 05:12:36 dist-test-slave-mr2f krb5kdc[17309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:36.368252 17322 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:12:36.368709 17322 webserver.cc:460] Webserver started at http://127.0.98.62:40927/ using document root <none> and password file <none>
I20260924 05:12:36.368927 17322 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:36.368993 17322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:36.369145 17322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:36.370199 17322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "8b052ff7d5f5487ab591a549f7a0b3cf"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "ff6a2762ef4b3880457b2282b7416c8c"
server_key_iv: "8b5cb125cb55dfa88bb6516b64253282"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.370579 17322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b052ff7d5f5487ab591a549f7a0b3cf"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "ff6a2762ef4b3880457b2282b7416c8c"
server_key_iv: "8b5cb125cb55dfa88bb6516b64253282"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.371922 17322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:12:36.372601 17337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.372768 17322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:36.372871 17322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "8b052ff7d5f5487ab591a549f7a0b3cf"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "ff6a2762ef4b3880457b2282b7416c8c"
server_key_iv: "8b5cb125cb55dfa88bb6516b64253282"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.372947 17322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:36.388293 17322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:36.390144 17322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:36.390341 17322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:36.394625 17322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41957
I20260924 05:12:36.394656 17389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41957 every 8 connection(s)
I20260924 05:12:36.395049 17322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:36.395900 17390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:36.398029 17390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf: Bootstrap starting.
I20260924 05:12:36.398546 17390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:36.398829 17390 log.cc:824] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:36.399478 17390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf: No bootstrap required, opened a new log
I20260924 05:12:36.400955 17390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b052ff7d5f5487ab591a549f7a0b3cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 05:12:36.401098 17390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:36.401119 17390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b052ff7d5f5487ab591a549f7a0b3cf, State: Initialized, Role: FOLLOWER
I20260924 05:12:36.401211 17390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b052ff7d5f5487ab591a549f7a0b3cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 05:12:36.401260 17390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:36.401280 17390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:36.401350 17390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:36.401734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 17322
I20260924 05:12:36.401850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:36.402076 392 external_mini_cluster.cc:1428] Setting key d5400d48c56112aa6f5108a89d6b46a6
I20260924 05:12:36.402081 17390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b052ff7d5f5487ab591a549f7a0b3cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 05:12:36.402174 17390 leader_election.cc:302] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b052ff7d5f5487ab591a549f7a0b3cf; no voters:
I20260924 05:12:36.402333 17390 leader_election.cc:288] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:36.402359 17393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:36.402534 17393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [term 1 LEADER]: Becoming Leader. State: Replica: 8b052ff7d5f5487ab591a549f7a0b3cf, State: Running, Role: LEADER
I20260924 05:12:36.402633 17390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:36.402662 17393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b052ff7d5f5487ab591a549f7a0b3cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 05:12:36.403195 17394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b052ff7d5f5487ab591a549f7a0b3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b052ff7d5f5487ab591a549f7a0b3cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } }
I20260924 05:12:36.403359 17394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:36.403702 17395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b052ff7d5f5487ab591a549f7a0b3cf. Latest consensus state: current_term: 1 leader_uuid: "8b052ff7d5f5487ab591a549f7a0b3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b052ff7d5f5487ab591a549f7a0b3cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } }
I20260924 05:12:36.403796 17400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:36.403829 17395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:36.404451 17400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:36.406698 17400 catalog_manager.cc:1347] Generated new cluster ID: 2375866aefa44bb79163808c15645f99
I20260924 05:12:36.406749 17400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:12:36 dist-test-slave-mr2f krb5kdc[17309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:36.413394 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.402549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.0.98.62:41957)
0924 05:12:36.402815 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.402825 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.402880 (+ 55us) server_negotiation.cc:407] Connection header received
0924 05:12:36.403038 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.403045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.403105 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.404073 (+ 968us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:36.404191 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.405574 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.406363 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.406995 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.407090 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.409838 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:36.409841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:36.409842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:36.409853 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:36.411787 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.412180 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.412183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.412183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.412295 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.412507 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.412508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.412508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.412784 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:36.412935 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.413043 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.413167 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":57,"threads_started":1}
I20260924 05:12:36.416632 17400 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:36.417099 17400 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:36.420421 17400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf: Generated new TSK 0
I20260924 05:12:36.420568 17400 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:12:36.445541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:36.524312 17416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:36.524540 17416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:36.524578 17416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:36.525723 17416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:36.525789 17416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:36.525835 17416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:12:36.527272 17416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:36.527554 17416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:36.527769 17416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:36.532516 17424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:36.532567 17421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:36.532670 17416 server_base.cc:1034] running on GCE node
W20260924 05:12:36.532513 17422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:36.532943 17416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:36.533192 17416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:36.534348 17416 hybrid_clock.cc:648] HybridClock initialized: now 1790226756534333 us; error 27 us; skew 500 ppm
Sep 24 05:12:36 dist-test-slave-mr2f krb5kdc[17309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:36.536736 17416 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:12:36.537148 17416 webserver.cc:460] Webserver started at http://127.0.98.1:33103/ using document root <none> and password file <none>
I20260924 05:12:36.537364 17416 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:36.537431 17416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:36.537576 17416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:36.538551 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "52a4b06087ef476fb0f8d1daa810a60f"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "996aed0443f33257d433744d7ca947a4"
server_key_iv: "74badb8208f688d2cabd688646cdf60a"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.538931 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "52a4b06087ef476fb0f8d1daa810a60f"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "996aed0443f33257d433744d7ca947a4"
server_key_iv: "74badb8208f688d2cabd688646cdf60a"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.540187 17416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:12:36.540858 17431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.541031 17416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.541123 17416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "52a4b06087ef476fb0f8d1daa810a60f"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "996aed0443f33257d433744d7ca947a4"
server_key_iv: "74badb8208f688d2cabd688646cdf60a"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.541203 17416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:36.549824 17416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:36.551558 17416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:36.551724 17416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:36.551949 17416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:36.552227 17416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:36.552279 17416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.552318 17416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:36.552346 17416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.557869 17416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38165
I20260924 05:12:36.557945 17544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38165 every 8 connection(s)
I20260924 05:12:36.558238 17416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:12:36.560122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 17416
I20260924 05:12:36.560233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:12:36.560448 392 external_mini_cluster.cc:1428] Setting key b340c72e69d9187dfe195e6756836d8e
Sep 24 05:12:36 dist-test-slave-mr2f krb5kdc[17309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:36.567754 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.558879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58263 (local address 127.0.98.62:41957)
0924 05:12:36.558917 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.558920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.559141 (+ 221us) server_negotiation.cc:407] Connection header received
0924 05:12:36.560042 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.560043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.560059 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.560122 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:36.560863 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.561450 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.562444 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.562666 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.562753 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.564925 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:36.564926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:36.564927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:36.564935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:36.566348 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.566789 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.566791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.566791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.566847 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.567106 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.567107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.567108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.567223 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:36.567293 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.567580 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.567667 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:12:36.567962 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.558897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.98.1:58263)
0924 05:12:36.559029 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.559060 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.559935 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.560188 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.560192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.560505 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:36.560769 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.560780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.561545 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.561546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.562346 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.562349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.562782 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.562784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.562931 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.563327 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:12:36.563335 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:36.564826 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:36.566458 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:36.566465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:36.566467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:36.566695 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:36.566929 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:36.566930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:36.566930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:36.567038 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:36.567295 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:36.567302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:36.567433 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.567528 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.567693 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260924 05:12:36.568435 17545 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41957
I20260924 05:12:36.568513 17545 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:36.568684 17545 heartbeater.cc:511] Master 127.0.98.62:41957 requested a full tablet report, sending...
I20260924 05:12:36.569195 17354 ts_manager.cc:194] Registered new tserver with Master: 52a4b06087ef476fb0f8d1daa810a60f (127.0.98.1:38165)
I20260924 05:12:36.569854 17354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:12:36.583796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:36.661908 17550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:36.662130 17550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:36.662168 17550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:36.663309 17550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:36.663381 17550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:36.663424 17550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:12:36.664804 17550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:36.665045 17550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:36.665242 17550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:36.669831 17555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:36.669831 17558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:36.670051 17550 server_base.cc:1034] running on GCE node
W20260924 05:12:36.669849 17556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:36.670281 17550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:36.670542 17550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:36.671710 17550 hybrid_clock.cc:648] HybridClock initialized: now 1790226756671692 us; error 30 us; skew 500 ppm
Sep 24 05:12:36 dist-test-slave-mr2f krb5kdc[17309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:36.673938 17550 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:12:36.674350 17550 webserver.cc:460] Webserver started at http://127.0.98.2:39859/ using document root <none> and password file <none>
I20260924 05:12:36.674525 17550 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:36.674575 17550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:36.674654 17550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:36.675664 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "e2ad5a51b189498e9e5da1c8f187ecd4"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "2c555186a29526489a2a00f9342f83b1"
server_key_iv: "f5236e2c69d678c47992f7e73266ff80"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.675978 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e2ad5a51b189498e9e5da1c8f187ecd4"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "2c555186a29526489a2a00f9342f83b1"
server_key_iv: "f5236e2c69d678c47992f7e73266ff80"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.677217 17550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:12:36.677830 17565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.678002 17550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:36.678076 17550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "e2ad5a51b189498e9e5da1c8f187ecd4"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "2c555186a29526489a2a00f9342f83b1"
server_key_iv: "f5236e2c69d678c47992f7e73266ff80"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.678125 17550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:36.694202 17550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:36.696086 17550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:36.696188 17550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:36.696354 17550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:36.696585 17550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:36.696614 17550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.696635 17550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:36.696647 17550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.702188 17550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44311
I20260924 05:12:36.702243 17678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44311 every 8 connection(s)
I20260924 05:12:36.702518 17550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:12:36.708036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 17550
I20260924 05:12:36.708123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:12:36.708331 392 external_mini_cluster.cc:1428] Setting key 067f7bac88bf0c62b0002ad31e05a99b
Sep 24 05:12:36 dist-test-slave-mr2f krb5kdc[17309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:36.712198 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.703276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60017 (local address 127.0.98.62:41957)
0924 05:12:36.703333 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.703335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.703531 (+ 196us) server_negotiation.cc:407] Connection header received
0924 05:12:36.704543 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.704544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.704556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.704601 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:36.705238 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.705727 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.706695 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.706961 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.707042 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.709189 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:36.709191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:36.709191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:36.709199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:36.710799 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.711185 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.711186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.711187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.711253 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.711465 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.711465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.711466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.711609 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:36.711684 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.711944 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.712085 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:12:36.712286 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.703239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.98.2:60017)
0924 05:12:36.703399 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.703437 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.704437 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.704680 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.704683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.704978 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:36.705145 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.705149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.705815 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.705816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.706592 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.706594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.707051 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.707052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.707177 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.707643 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:12:36.707652 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:36.709096 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:36.710882 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:36.710888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:36.710889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:36.711108 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:36.711320 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:36.711320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:36.711321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:36.711411 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:36.711679 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:36.711683 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:36.711788 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.711900 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.712071 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 05:12:36.720566 17679 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41957
I20260924 05:12:36.720667 17679 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:36.720852 17679 heartbeater.cc:511] Master 127.0.98.62:41957 requested a full tablet report, sending...
I20260924 05:12:36.721324 17354 ts_manager.cc:194] Registered new tserver with Master: e2ad5a51b189498e9e5da1c8f187ecd4 (127.0.98.2:44311)
I20260924 05:12:36.721812 17354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:12:36.733789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:36.816179 17684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:36.816365 17684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:36.816387 17684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:36.817458 17684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:36.817492 17684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:36.817516 17684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:12:36.818985 17684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:36.819159 17684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:36.819314 17684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:36.824034 17692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:36.824013 17690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:36.824012 17689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:36.824254 17684 server_base.cc:1034] running on GCE node
I20260924 05:12:36.824538 17684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:36.824759 17684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:36.825901 17684 hybrid_clock.cc:648] HybridClock initialized: now 1790226756825888 us; error 29 us; skew 500 ppm
Sep 24 05:12:36 dist-test-slave-mr2f krb5kdc[17309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:36.828255 17684 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:12:36.828655 17684 webserver.cc:460] Webserver started at http://127.0.98.3:40345/ using document root <none> and password file <none>
I20260924 05:12:36.828824 17684 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:36.828868 17684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:36.828949 17684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:36.829866 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "e909c15368fa4f458522bd37cf724c64"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "baf7894a2ae5c9b97dae8125bbc6bc5d"
server_key_iv: "7ac72707122142fc8705607246a05367"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.830147 17684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e909c15368fa4f458522bd37cf724c64"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "baf7894a2ae5c9b97dae8125bbc6bc5d"
server_key_iv: "7ac72707122142fc8705607246a05367"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.831389 17684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:12:36.832024 17699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.832192 17684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:36.832288 17684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "e909c15368fa4f458522bd37cf724c64"
format_stamp: "Formatted at 2026-09-24 05:12:36 on dist-test-slave-mr2f"
server_key: "baf7894a2ae5c9b97dae8125bbc6bc5d"
server_key_iv: "7ac72707122142fc8705607246a05367"
server_key_version: "encryptionkey@0"
I20260924 05:12:36.832365 17684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:36.840034 17684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:36.841739 17684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:36.841892 17684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:36.842092 17684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:36.842363 17684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:36.842415 17684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.842453 17684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:36.842481 17684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:36.848383 17684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40211
I20260924 05:12:36.848428 17812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40211 every 8 connection(s)
I20260924 05:12:36.848695 17684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:12:36 dist-test-slave-mr2f krb5kdc[17309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:36.857144 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.849336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45521 (local address 127.0.98.62:41957)
0924 05:12:36.849400 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.849402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.849570 (+ 168us) server_negotiation.cc:407] Connection header received
0924 05:12:36.850449 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.850450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.850459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.850508 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:36.851072 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.851537 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.852486 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.852658 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.852706 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.854565 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:36.854566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:36.854566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:36.854572 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:36.855933 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.856269 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.856270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.856270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.856317 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.856520 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.856521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.856521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.856613 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:36.856664 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.857031 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.857086 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 05:12:36.857380 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.849298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.98.3:45521)
0924 05:12:36.849445 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.849483 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.850361 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.850564 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.850567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.850837 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:36.851006 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.851010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.851610 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.851611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.852414 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.852416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.852744 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.852745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.852838 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.853269 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:12:36.853277 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:36.854485 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:36.856001 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:36.856007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:36.856008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:36.856204 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:36.856373 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:36.856374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:36.856374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:36.856479 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:36.856777 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:36.856783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:36.856892 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.857007 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.857117 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260924 05:12:36.857969 17813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41957
I20260924 05:12:36.858065 17813 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:36.858129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 17684
I20260924 05:12:36.858242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:12:36.858283 17813 heartbeater.cc:511] Master 127.0.98.62:41957 requested a full tablet report, sending...
I20260924 05:12:36.858464 392 external_mini_cluster.cc:1428] Setting key 90dda36000cfe3935784ab0f91ec9677
I20260924 05:12:36.858690 17354 ts_manager.cc:194] Registered new tserver with Master: e909c15368fa4f458522bd37cf724c64 (127.0.98.3:40211)
I20260924 05:12:36.859198 17354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45521
I20260924 05:12:36.859337 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:12:36.866982 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.860543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.98.62:41957)
0924 05:12:36.860613 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.860615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.860680 (+ 65us) server_negotiation.cc:407] Connection header received
0924 05:12:36.860729 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.860730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.860737 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.860774 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:36.861272 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.861712 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.862379 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.862554 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.862605 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.864005 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:36.864006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:36.864007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:36.864014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:36.865688 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.866037 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.866038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.866038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.866111 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:36.866370 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:36.866371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:36.866371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:36.866556 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:36.866667 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.866838 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.866917 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 05:12:36.868538 17354 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:12:36.869984 17354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:12:36.876127 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.871966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38165 (local address 127.0.0.1:47652)
0924 05:12:36.872280 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.872322 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.872397 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.872644 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.872647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.872651 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:36.872799 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.872801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.873685 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.873686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.874767 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.874770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.875797 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.875798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.875908 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.875918 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.875996 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.876047 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":53,"threads_started":1}
I20260924 05:12:36.876137 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.872072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.0.98.1:38165)
0924 05:12:36.872326 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.872331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.872482 (+ 151us) server_negotiation.cc:407] Connection header received
0924 05:12:36.872529 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.872531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.872546 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.872626 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:36.872921 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.873595 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.874895 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.875651 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.875684 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.875735 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.876012 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.876049 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":107,"threads_started":1}
I20260924 05:12:36.876544 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.871970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44311 (local address 127.0.0.1:56790)
0924 05:12:36.872304 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.872334 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.872397 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.872802 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.872803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.872805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:36.872873 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.872876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.874175 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.874176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.875400 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.875403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.876287 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.876288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.876345 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.876348 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.876411 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.876447 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":99,"threads_started":1}
I20260924 05:12:36.876564 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.872067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.0.98.2:44311)
0924 05:12:36.872552 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.872556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.872566 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:12:36.872603 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.872605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.872619 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.872711 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:36.872949 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.874067 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.875479 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.876209 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.876283 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.876334 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.876448 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.876486 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":91,"threads_started":1}
I20260924 05:12:36.876643 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.872290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.0.98.3:40211)
0924 05:12:36.872779 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.872783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.872792 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:12:36.872824 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.872825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.872842 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.873050 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:36.873117 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.874153 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.875309 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.876085 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.876162 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.876500 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.876523 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.876558 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":94,"threads_started":1}
I20260924 05:12:36.876768 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.872177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40211 (local address 127.0.0.1:60172)
0924 05:12:36.872378 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.872387 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.872429 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.872952 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.872953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.872954 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:36.872983 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.872985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.874244 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.874245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.875224 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.875226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.876266 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.876267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.876327 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.876332 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.876373 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.876415 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":67,"threads_started":1}
I20260924 05:12:36.877223 17479 tablet_service.cc:1467] Processing CreateTablet for tablet dd2ac78239294366afbd125faa7ec098 (DEFAULT_TABLE table=test-table [id=535cbf8df12d49e594d39bb6c15b4847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:36.877578 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2ac78239294366afbd125faa7ec098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:36.877521 17613 tablet_service.cc:1467] Processing CreateTablet for tablet dd2ac78239294366afbd125faa7ec098 (DEFAULT_TABLE table=test-table [id=535cbf8df12d49e594d39bb6c15b4847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:36.877818 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2ac78239294366afbd125faa7ec098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:36.879148 17747 tablet_service.cc:1467] Processing CreateTablet for tablet dd2ac78239294366afbd125faa7ec098 (DEFAULT_TABLE table=test-table [id=535cbf8df12d49e594d39bb6c15b4847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:36.879443 17747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd2ac78239294366afbd125faa7ec098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:36.881035 17828 tablet_bootstrap.cc:492] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f: Bootstrap starting.
I20260924 05:12:36.881775 17828 tablet_bootstrap.cc:654] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:36.882380 17828 log.cc:824] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:36.882735 17827 tablet_bootstrap.cc:492] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: Bootstrap starting.
I20260924 05:12:36.882982 17830 tablet_bootstrap.cc:492] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: Bootstrap starting.
I20260924 05:12:36.883235 17828 tablet_bootstrap.cc:492] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f: No bootstrap required, opened a new log
I20260924 05:12:36.883291 17828 ts_tablet_manager.cc:1397] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:12:36.883566 17830 tablet_bootstrap.cc:654] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:36.884033 17830 log.cc:824] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:36.884685 17830 tablet_bootstrap.cc:492] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: No bootstrap required, opened a new log
I20260924 05:12:36.884765 17830 ts_tablet_manager.cc:1397] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:12:36.884984 17828 raft_consensus.cc:348] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.885113 17828 raft_consensus.cc:374] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:36.885135 17828 raft_consensus.cc:729] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a4b06087ef476fb0f8d1daa810a60f, State: Initialized, Role: FOLLOWER
I20260924 05:12:36.885277 17828 consensus_queue.cc:260] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.885583 17545 heartbeater.cc:503] Master 127.0.98.62:41957 was elected leader, sending a full tablet report...
I20260924 05:12:36.885567 17828 ts_tablet_manager.cc:1428] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:12:36.886155 17827 tablet_bootstrap.cc:654] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:36.886714 17827 log.cc:824] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:36.886749 17830 raft_consensus.cc:348] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.887001 17830 raft_consensus.cc:374] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:36.887060 17830 raft_consensus.cc:729] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e909c15368fa4f458522bd37cf724c64, State: Initialized, Role: FOLLOWER
I20260924 05:12:36.887183 17830 consensus_queue.cc:260] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.887476 17830 ts_tablet_manager.cc:1428] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:12:36.887507 17813 heartbeater.cc:503] Master 127.0.98.62:41957 was elected leader, sending a full tablet report...
I20260924 05:12:36.887526 17827 tablet_bootstrap.cc:492] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: No bootstrap required, opened a new log
I20260924 05:12:36.887598 17827 ts_tablet_manager.cc:1397] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260924 05:12:36.889183 17827 raft_consensus.cc:348] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.889319 17827 raft_consensus.cc:374] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:36.889340 17827 raft_consensus.cc:729] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ad5a51b189498e9e5da1c8f187ecd4, State: Initialized, Role: FOLLOWER
I20260924 05:12:36.889477 17827 consensus_queue.cc:260] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.889773 17679 heartbeater.cc:503] Master 127.0.98.62:41957 was elected leader, sending a full tablet report...
I20260924 05:12:36.889794 17827 ts_tablet_manager.cc:1428] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 05:12:36.939584 17832 raft_consensus.cc:482] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:12:36.939698 17832 raft_consensus.cc:504] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.940003 17832 leader_election.cc:288] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2ad5a51b189498e9e5da1c8f187ecd4 (127.0.98.2:44311), e909c15368fa4f458522bd37cf724c64 (127.0.98.3:40211)
I20260924 05:12:36.940672 17834 raft_consensus.cc:482] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:12:36.940788 17834 raft_consensus.cc:504] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.941280 17834 leader_election.cc:288] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2ad5a51b189498e9e5da1c8f187ecd4 (127.0.98.2:44311), 52a4b06087ef476fb0f8d1daa810a60f (127.0.98.1:38165)
I20260924 05:12:36.943969 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.940173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44311 (local address 127.0.98.1:48643)
0924 05:12:36.940231 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.940285 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.940342 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.940508 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.940509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.940511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:36.940592 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.940594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.941553 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.941554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.942937 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.942940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.943710 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.943711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.943791 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.943794 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.943845 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.943888 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":123}
I20260924 05:12:36.943974 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.940282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48643 (local address 127.0.98.2:44311)
0924 05:12:36.940351 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.940353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.940372 (+ 19us) server_negotiation.cc:407] Connection header received
0924 05:12:36.940399 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.940400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.940412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.940480 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:36.940673 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.941458 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.943025 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.943594 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.943656 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.943668 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.943852 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.943895 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 05:12:36.944375 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.940361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33805 (local address 127.0.98.3:40211)
0924 05:12:36.940404 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.940406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.940492 (+ 86us) server_negotiation.cc:407] Connection header received
0924 05:12:36.940556 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.940557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.940570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.940628 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:36.940783 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.941457 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.943400 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.943932 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.943989 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.943998 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.944284 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.944318 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":23}
I20260924 05:12:36.944514 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2ac78239294366afbd125faa7ec098" candidate_uuid: "52a4b06087ef476fb0f8d1daa810a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" is_pre_election: true
I20260924 05:12:36.944676 17633 raft_consensus.cc:2454] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a4b06087ef476fb0f8d1daa810a60f in term 0.
I20260924 05:12:36.944713 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.940282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40211 (local address 127.0.98.1:33805)
0924 05:12:36.940426 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.940435 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.940484 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.940629 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.940631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.940633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:36.940702 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.940703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.941553 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.941554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.943321 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.943323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.944107 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.944108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.944171 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.944175 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.944588 (+ 413us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.944626 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":45,"thread_start_us":85,"threads_started":1}
I20260924 05:12:36.944806 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.941246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38165 (local address 127.0.98.3:58803)
0924 05:12:36.941422 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.941432 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.941473 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.941736 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.941737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.941739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:36.941797 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.941799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.942683 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.942684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.943871 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.943874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.944593 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.944594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.944656 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.944659 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.944703 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.944741 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":46}
I20260924 05:12:36.944936 17434 leader_election.cc:302] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a4b06087ef476fb0f8d1daa810a60f, e2ad5a51b189498e9e5da1c8f187ecd4; no voters:
I20260924 05:12:36.945060 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.941575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58803 (local address 127.0.98.1:38165)
0924 05:12:36.941610 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.941611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.941618 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:12:36.941638 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.941638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.941648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.941698 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:36.941873 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.942603 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.943971 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.944501 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.944557 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.944564 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.944887 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.944926 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:12:36.945274 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2ac78239294366afbd125faa7ec098" candidate_uuid: "52a4b06087ef476fb0f8d1daa810a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e909c15368fa4f458522bd37cf724c64" is_pre_election: true
I20260924 05:12:36.945372 17767 raft_consensus.cc:2454] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a4b06087ef476fb0f8d1daa810a60f in term 0.
I20260924 05:12:36.945447 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.941438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35271 (local address 127.0.98.2:44311)
0924 05:12:36.941597 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.941599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.941722 (+ 123us) server_negotiation.cc:407] Connection header received
0924 05:12:36.941812 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.941813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.941822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.941877 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:36.942056 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.942755 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.944415 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.945051 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.945117 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.945127 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.945331 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.945371 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260924 05:12:36.945475 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.941246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44311 (local address 127.0.98.3:35271)
0924 05:12:36.941669 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.941678 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:36.941743 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:36.941936 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:36.941936 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:36.941938 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:36.941987 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.941988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.942862 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.942863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.944321 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:36.944323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.945170 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.945171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:36.945252 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.945255 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:36.945317 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 05:12:36.945351 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":38,"thread_start_us":99,"threads_started":1}
I20260924 05:12:36.945494 17832 raft_consensus.cc:2781] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:12:36.945561 17832 raft_consensus.cc:482] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:12:36.945530 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2ac78239294366afbd125faa7ec098" candidate_uuid: "e909c15368fa4f458522bd37cf724c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a4b06087ef476fb0f8d1daa810a60f" is_pre_election: true
I20260924 05:12:36.945591 17832 raft_consensus.cc:3037] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:36.945960 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2ac78239294366afbd125faa7ec098" candidate_uuid: "e909c15368fa4f458522bd37cf724c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" is_pre_election: true
I20260924 05:12:36.946035 17633 raft_consensus.cc:2454] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e909c15368fa4f458522bd37cf724c64 in term 0.
I20260924 05:12:36.946208 17702 leader_election.cc:302] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2ad5a51b189498e9e5da1c8f187ecd4, e909c15368fa4f458522bd37cf724c64; no voters:
I20260924 05:12:36.946353 17834 raft_consensus.cc:2781] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:12:36.946431 17834 raft_consensus.cc:482] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:12:36.946465 17834 raft_consensus.cc:3037] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:36.946556 17832 raft_consensus.cc:504] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.946692 17832 leader_election.cc:288] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [CANDIDATE]: Term 1 election: Requested vote from peers e2ad5a51b189498e9e5da1c8f187ecd4 (127.0.98.2:44311), e909c15368fa4f458522bd37cf724c64 (127.0.98.3:40211)
I20260924 05:12:36.946691 17499 raft_consensus.cc:2379] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e909c15368fa4f458522bd37cf724c64 in current term 1: Already voted for candidate 52a4b06087ef476fb0f8d1daa810a60f in this term.
I20260924 05:12:36.946897 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2ac78239294366afbd125faa7ec098" candidate_uuid: "52a4b06087ef476fb0f8d1daa810a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4"
I20260924 05:12:36.946894 17767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2ac78239294366afbd125faa7ec098" candidate_uuid: "52a4b06087ef476fb0f8d1daa810a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e909c15368fa4f458522bd37cf724c64"
I20260924 05:12:36.946962 17633 raft_consensus.cc:3037] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:36.947707 17834 raft_consensus.cc:504] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.947817 17633 raft_consensus.cc:2454] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a4b06087ef476fb0f8d1daa810a60f in term 1.
I20260924 05:12:36.947870 17767 raft_consensus.cc:2379] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 52a4b06087ef476fb0f8d1daa810a60f in current term 1: Already voted for candidate e909c15368fa4f458522bd37cf724c64 in this term.
I20260924 05:12:36.947954 17633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2ac78239294366afbd125faa7ec098" candidate_uuid: "e909c15368fa4f458522bd37cf724c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4"
I20260924 05:12:36.947978 17434 leader_election.cc:302] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a4b06087ef476fb0f8d1daa810a60f, e2ad5a51b189498e9e5da1c8f187ecd4; no voters:
I20260924 05:12:36.948024 17633 raft_consensus.cc:2379] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e909c15368fa4f458522bd37cf724c64 in current term 1: Already voted for candidate 52a4b06087ef476fb0f8d1daa810a60f in this term.
I20260924 05:12:36.948033 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd2ac78239294366afbd125faa7ec098" candidate_uuid: "e909c15368fa4f458522bd37cf724c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a4b06087ef476fb0f8d1daa810a60f"
I20260924 05:12:36.948108 17499 raft_consensus.cc:2379] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e909c15368fa4f458522bd37cf724c64 in current term 1: Already voted for candidate 52a4b06087ef476fb0f8d1daa810a60f in this term.
I20260924 05:12:36.948171 17832 raft_consensus.cc:2781] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:36.948258 17703 leader_election.cc:302] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e909c15368fa4f458522bd37cf724c64; no voters: 52a4b06087ef476fb0f8d1daa810a60f, e2ad5a51b189498e9e5da1c8f187ecd4
I20260924 05:12:36.948328 17832 raft_consensus.cc:686] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [term 1 LEADER]: Becoming Leader. State: Replica: 52a4b06087ef476fb0f8d1daa810a60f, State: Running, Role: LEADER
I20260924 05:12:36.948400 17834 leader_election.cc:288] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [CANDIDATE]: Term 1 election: Requested vote from peers e2ad5a51b189498e9e5da1c8f187ecd4 (127.0.98.2:44311), 52a4b06087ef476fb0f8d1daa810a60f (127.0.98.1:38165)
I20260924 05:12:36.948443 17834 raft_consensus.cc:2726] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 05:12:36.948414 17832 consensus_queue.cc:237] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } }
I20260924 05:12:36.949113 17353 catalog_manager.cc:5495] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a4b06087ef476fb0f8d1daa810a60f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "52a4b06087ef476fb0f8d1daa810a60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52a4b06087ef476fb0f8d1daa810a60f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 } health_report { overall_health: UNKNOWN } } }
W20260924 05:12:36.953047 17680 tablet.cc:2369] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:12:36.969986 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.967809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.0.98.1:38165)
0924 05:12:36.967865 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:36.967867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:36.967875 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:12:36.967908 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:36.967912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:36.967929 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:36.967986 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:12:36.968102 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.968546 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.969288 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:36.969535 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:36.969606 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:36.969734 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:12:36.969826 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:12:36.969890 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:36.969920 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 05:12:36.969937 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:12:36.971683 17633 raft_consensus.cc:1261] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 1 FOLLOWER]: Refusing update from remote peer 52a4b06087ef476fb0f8d1daa810a60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:36.971670 17767 raft_consensus.cc:1261] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 1 FOLLOWER]: Refusing update from remote peer 52a4b06087ef476fb0f8d1daa810a60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:36.971997 17832 consensus_queue.cc:1035] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ad5a51b189498e9e5da1c8f187ecd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:36.972110 17839 consensus_queue.cc:1035] T dd2ac78239294366afbd125faa7ec098 P 52a4b06087ef476fb0f8d1daa810a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e909c15368fa4f458522bd37cf724c64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:36.974519 17844 mvcc.cc:205] Tried to move back new op lower bound from 7332768796553519104 to 7332768796461150208. Current Snapshot: MvccSnapshot[applied={T|T < 7332768796553519104}]
I20260924 05:12:36.977197 17353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:36.977286 17353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:36.978008 17353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b052ff7d5f5487ab591a549f7a0b3cf: Sending DeleteTablet for 3 replicas of tablet dd2ac78239294366afbd125faa7ec098
I20260924 05:12:36.978374 17613 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2ac78239294366afbd125faa7ec098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56790
I20260924 05:12:36.978538 17479 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2ac78239294366afbd125faa7ec098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47652
I20260924 05:12:36.978612 17747 tablet_service.cc:1514] Processing DeleteTablet for tablet dd2ac78239294366afbd125faa7ec098 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60172
I20260924 05:12:36.978820 17854 tablet_replica.cc:330] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: stopping tablet replica
I20260924 05:12:36.978904 17852 tablet_replica.cc:330] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: stopping tablet replica
I20260924 05:12:36.978991 17854 raft_consensus.cc:2229] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:12:36.979043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 17416
I20260924 05:12:36.979063 17852 raft_consensus.cc:2229] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:12:36.979596 17852 raft_consensus.cc:2258] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:36.980286 17852 ts_tablet_manager.cc:1905] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:36.981281 17854 raft_consensus.cc:2258] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:36.981802 17854 ts_tablet_manager.cc:1905] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:36.981866 17852 ts_tablet_manager.cc:1918] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:36.981961 17852 log.cc:1196] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/dd2ac78239294366afbd125faa7ec098
I20260924 05:12:36.982467 17852 ts_tablet_manager.cc:1939] T dd2ac78239294366afbd125faa7ec098 P e2ad5a51b189498e9e5da1c8f187ecd4: Deleting consensus metadata
I20260924 05:12:36.982882 17340 catalog_manager.cc:4841] TS e2ad5a51b189498e9e5da1c8f187ecd4 (127.0.98.2:44311): tablet dd2ac78239294366afbd125faa7ec098 (table test-table [id=535cbf8df12d49e594d39bb6c15b4847]) successfully deleted
I20260924 05:12:36.983647 17854 ts_tablet_manager.cc:1918] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:36.983717 17854 log.cc:1196] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/dd2ac78239294366afbd125faa7ec098
I20260924 05:12:36.983975 17854 ts_tablet_manager.cc:1939] T dd2ac78239294366afbd125faa7ec098 P e909c15368fa4f458522bd37cf724c64: Deleting consensus metadata
I20260924 05:12:36.984688 17340 catalog_manager.cc:4841] TS e909c15368fa4f458522bd37cf724c64 (127.0.98.3:40211): tablet dd2ac78239294366afbd125faa7ec098 (table test-table [id=535cbf8df12d49e594d39bb6c15b4847]) successfully deleted
W20260924 05:12:36.984779 17341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38165 (error 108)
I20260924 05:12:36.985018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 17550
I20260924 05:12:36.985596 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:36.985118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38165 (local address 127.0.0.1:47676)
0924 05:12:36.985532 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:36.985572 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397}
W20260924 05:12:36.985739 17341 catalog_manager.cc:4584] TS 52a4b06087ef476fb0f8d1daa810a60f (127.0.98.1:38165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd2ac78239294366afbd125faa7ec098: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38165: connect: Connection refused (error 111)
I20260924 05:12:36.990082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 17684
I20260924 05:12:36.994561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 17322
2026-09-24T05:12:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1858 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T05:12:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:37Z Disabled control of system clock
I20260924 05:12:37.021347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39641
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:12:37.098526 17859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:37.098730 17859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:37.098750 17859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:37.099860 17859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:37.099891 17859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:37.099905 17859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:37.099916 17859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:37.099927 17859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:37.101089 17859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:37.101253 17859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:37.101452 17859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:37.105997 17864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:37.106081 17859 server_base.cc:1034] running on GCE node
W20260924 05:12:37.106006 17867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:37.106011 17865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:37.106482 17859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:37.106747 17859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:37.107963 17859 hybrid_clock.cc:648] HybridClock initialized: now 1790226757107952 us; error 30 us; skew 500 ppm
I20260924 05:12:37.109112 17859 webserver.cc:460] Webserver started at http://127.0.98.62:43391/ using document root <none> and password file <none>
I20260924 05:12:37.109329 17859 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:37.109393 17859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:37.109536 17859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:37.110550 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "5d208c8dfd9940d7b1f4973305ecfb3c"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "38a4114c87671c630152a941e93a6054"
server_key_iv: "d9c8be6c16d6be6a41ff9b63870d0070"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.110922 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d208c8dfd9940d7b1f4973305ecfb3c"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "38a4114c87671c630152a941e93a6054"
server_key_iv: "d9c8be6c16d6be6a41ff9b63870d0070"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.112154 17859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:12:37.112821 17873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.113001 17859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:12:37.113091 17859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "5d208c8dfd9940d7b1f4973305ecfb3c"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "38a4114c87671c630152a941e93a6054"
server_key_iv: "d9c8be6c16d6be6a41ff9b63870d0070"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.113188 17859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:37.126551 17859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:37.126863 17859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:37.127028 17859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:37.130684 17859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39641
I20260924 05:12:37.130724 17925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39641 every 8 connection(s)
I20260924 05:12:37.131104 17859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:37.131994 17926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:37.134107 17926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c: Bootstrap starting.
I20260924 05:12:37.134706 17926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:37.135047 17926 log.cc:824] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:37.135110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 17859
I20260924 05:12:37.135246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:37.135488 392 external_mini_cluster.cc:1428] Setting key 128e3b66ad4d36492b78836bc3104a7e
I20260924 05:12:37.135823 17926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c: No bootstrap required, opened a new log
I20260924 05:12:37.138543 17926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d208c8dfd9940d7b1f4973305ecfb3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } }
I20260924 05:12:37.138700 17926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:37.138728 17926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d208c8dfd9940d7b1f4973305ecfb3c, State: Initialized, Role: FOLLOWER
I20260924 05:12:37.138859 17926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d208c8dfd9940d7b1f4973305ecfb3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } }
I20260924 05:12:37.138933 17926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:37.138963 17926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:37.139007 17926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:37.139528 17926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d208c8dfd9940d7b1f4973305ecfb3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } }
I20260924 05:12:37.139611 17926 leader_election.cc:302] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d208c8dfd9940d7b1f4973305ecfb3c; no voters:
I20260924 05:12:37.139787 17926 leader_election.cc:288] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:37.139829 17931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:37.139978 17931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [term 1 LEADER]: Becoming Leader. State: Replica: 5d208c8dfd9940d7b1f4973305ecfb3c, State: Running, Role: LEADER
I20260924 05:12:37.140056 17926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:37.140058 17931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d208c8dfd9940d7b1f4973305ecfb3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } }
I20260924 05:12:37.141791 17933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d208c8dfd9940d7b1f4973305ecfb3c. Latest consensus state: current_term: 1 leader_uuid: "5d208c8dfd9940d7b1f4973305ecfb3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d208c8dfd9940d7b1f4973305ecfb3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } } }
I20260924 05:12:37.141966 17932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d208c8dfd9940d7b1f4973305ecfb3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d208c8dfd9940d7b1f4973305ecfb3c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } } }
I20260924 05:12:37.142097 17933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:37.142107 17932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:37.142127 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.136018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.0.98.62:39641)
0924 05:12:37.136236 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.136244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.136278 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:12:37.137104 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.137113 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.137186 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.137337 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:37.137505 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.138987 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.139889 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.140505 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.140626 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.141311 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:37.141315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:37.141315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:37.141524 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:37.141566 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:37.141579 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:37.141640 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.141842 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.141922 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
W20260924 05:12:37.142712 17945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:12:37.142762 17945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:12:37.142835 17946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:37.143033 17946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:37.144410 17946 catalog_manager.cc:1347] Generated new cluster ID: 7b212381124b41dbbaf33997546390e1
I20260924 05:12:37.144469 17946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:37.149921 17946 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:37.150547 17946 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:37.156673 17946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c: Generated new TSK 0
I20260924 05:12:37.156847 17946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:37.161487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39641
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:12:37.248804 17950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:37.249028 17950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:37.249064 17950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:37.250182 17950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:37.250243 17950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:37.250284 17950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:12:37.251646 17950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:37.251888 17950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:37.252158 17950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:37.257885 17956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:37.257879 17955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:37.258049 17950 server_base.cc:1034] running on GCE node
W20260924 05:12:37.257879 17958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:37.258323 17950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:37.258589 17950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:37.259775 17950 hybrid_clock.cc:648] HybridClock initialized: now 1790226757259738 us; error 51 us; skew 500 ppm
I20260924 05:12:37.261267 17950 webserver.cc:460] Webserver started at http://127.0.98.1:37855/ using document root <none> and password file <none>
I20260924 05:12:37.261543 17950 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:37.261606 17950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:37.261732 17950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:37.263087 17950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "643368f03a0445869f6b7ee59515684b"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "08bedef9dd29d947e9d016469fee2f58"
server_key_iv: "6ca22721496b1ebbd8a5b6174783b21f"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.263504 17950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "643368f03a0445869f6b7ee59515684b"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "08bedef9dd29d947e9d016469fee2f58"
server_key_iv: "6ca22721496b1ebbd8a5b6174783b21f"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.265098 17950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:12:37.265889 17964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.266078 17950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:37.266170 17950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "643368f03a0445869f6b7ee59515684b"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "08bedef9dd29d947e9d016469fee2f58"
server_key_iv: "6ca22721496b1ebbd8a5b6174783b21f"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.266238 17950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:37.279995 17950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:37.280257 17950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:37.280413 17950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:37.280619 17950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:37.280926 17950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:37.280958 17950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.281005 17950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:37.281029 17950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.286542 17950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41021
I20260924 05:12:37.286628 18077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41021 every 8 connection(s)
I20260924 05:12:37.286931 17950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:12:37.291720 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.287595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56891 (local address 127.0.98.62:39641)
0924 05:12:37.287654 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.287656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.287887 (+ 231us) server_negotiation.cc:407] Connection header received
0924 05:12:37.288378 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.288379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.288395 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.288456 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:37.289071 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.289523 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.290431 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.290625 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.290690 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.291456 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:37.291457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:37.291458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:37.291462 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:37.291475 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:37.291479 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:37.291523 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.291610 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.291633 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:12:37.291975 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.287581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39641 (local address 127.0.98.1:56891)
0924 05:12:37.287773 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.287803 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:37.288269 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:37.288543 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:37.288546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:37.288553 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:37.288976 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.288980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.289608 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.289609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.290310 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.290313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.290709 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.290710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.290891 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.291321 (+ 430us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:12:37.291322 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:37.291345 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:12:37.291365 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:37.291529 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:37.291533 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:37.291536 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:37.291611 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:12:37.291722 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260924 05:12:37.292508 18078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39641
I20260924 05:12:37.292598 18078 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:37.292800 18078 heartbeater.cc:511] Master 127.0.98.62:39641 requested a full tablet report, sending...
I20260924 05:12:37.293289 17890 ts_manager.cc:194] Registered new tserver with Master: 643368f03a0445869f6b7ee59515684b (127.0.98.1:41021)
I20260924 05:12:37.293828 17890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56891
I20260924 05:12:37.296836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 17950
I20260924 05:12:37.296959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:12:37.297169 392 external_mini_cluster.cc:1428] Setting key 2294f4d3f703f36dc3fa3c6cb5c40572
I20260924 05:12:37.297938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39641
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:12:37.375119 18081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:37.375298 18081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:37.375319 18081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:37.376441 18081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:37.376482 18081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:37.376530 18081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:12:37.377849 18081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:37.378086 18081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:37.378357 18081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:37.383082 18089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:37.383059 18086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:37.383077 18087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:37.383126 18081 server_base.cc:1034] running on GCE node
I20260924 05:12:37.383397 18081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:37.383649 18081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:37.384802 18081 hybrid_clock.cc:648] HybridClock initialized: now 1790226757384795 us; error 37 us; skew 500 ppm
I20260924 05:12:37.385933 18081 webserver.cc:460] Webserver started at http://127.0.98.2:33015/ using document root <none> and password file <none>
I20260924 05:12:37.386175 18081 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:37.386240 18081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:37.386365 18081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:37.387404 18081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "af458ffb9ad1400d8e660c1f3e19fb3b"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "e43d550bba6b82970e9470ed8c69576c"
server_key_iv: "71fdef04496a6c7abab8df66785dba40"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.387739 18081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af458ffb9ad1400d8e660c1f3e19fb3b"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "e43d550bba6b82970e9470ed8c69576c"
server_key_iv: "71fdef04496a6c7abab8df66785dba40"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.388945 18081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:12:37.389590 18095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.389737 18081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:37.389828 18081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "af458ffb9ad1400d8e660c1f3e19fb3b"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "e43d550bba6b82970e9470ed8c69576c"
server_key_iv: "71fdef04496a6c7abab8df66785dba40"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.389910 18081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:37.397500 18081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:37.397749 18081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:37.397891 18081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:37.398094 18081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:37.398389 18081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:37.398460 18081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.398499 18081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:37.398525 18081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.404081 18081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40769
I20260924 05:12:37.404121 18208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40769 every 8 connection(s)
I20260924 05:12:37.404410 18081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:12:37.409068 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.405093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33993 (local address 127.0.98.62:39641)
0924 05:12:37.405179 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.405181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.405417 (+ 236us) server_negotiation.cc:407] Connection header received
0924 05:12:37.405836 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.405838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.405848 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.405900 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:37.406480 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.407034 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.407921 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.408096 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.408151 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.408831 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:37.408832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:37.408833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:37.408836 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:37.408849 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:37.408852 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:37.408892 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.408990 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.409011 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 05:12:37.409260 18211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.405080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39641 (local address 127.0.98.2:33993)
0924 05:12:37.405285 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.405329 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:37.405755 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:37.405949 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:37.405952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:37.405959 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:37.406410 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.406414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.407106 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.407106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.407829 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.407830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.408167 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.408168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.408269 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.408713 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:12:37.408713 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:37.408737 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:12:37.408756 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:37.408904 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:37.408907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:37.408927 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 05:12:37.408977 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:12:37.409046 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":113,"threads_started":1}
I20260924 05:12:37.409785 18209 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39641
I20260924 05:12:37.409861 18209 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:37.409987 18209 heartbeater.cc:511] Master 127.0.98.62:39641 requested a full tablet report, sending...
I20260924 05:12:37.410312 17890 ts_manager.cc:194] Registered new tserver with Master: af458ffb9ad1400d8e660c1f3e19fb3b (127.0.98.2:40769)
I20260924 05:12:37.410674 17890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33993
I20260924 05:12:37.412297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 18081
I20260924 05:12:37.412417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:12:37.412607 392 external_mini_cluster.cc:1428] Setting key ce177f219041a8bd24be5ac7a6437d46
I20260924 05:12:37.413455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39641
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:12:37.491766 18212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:37.491945 18212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:37.491966 18212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:37.493125 18212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:37.493160 18212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:37.493222 18212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:12:37.494465 18212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:37.494632 18212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:37.494903 18212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:37.499439 18220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:37.499454 18217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:37.499456 18218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:37.499733 18212 server_base.cc:1034] running on GCE node
I20260924 05:12:37.499957 18212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:37.500202 18212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:37.501348 18212 hybrid_clock.cc:648] HybridClock initialized: now 1790226757501337 us; error 23 us; skew 500 ppm
I20260924 05:12:37.502507 18212 webserver.cc:460] Webserver started at http://127.0.98.3:41629/ using document root <none> and password file <none>
I20260924 05:12:37.502743 18212 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:37.502852 18212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:37.502995 18212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:37.503976 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "b7c4133b55be495c9d85df5a0f98812c"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "2688739350be10f86ea15e93db67acbd"
server_key_iv: "fc340b2d2d176cc06bcfe6ca9a2ae097"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.504317 18212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b7c4133b55be495c9d85df5a0f98812c"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "2688739350be10f86ea15e93db67acbd"
server_key_iv: "fc340b2d2d176cc06bcfe6ca9a2ae097"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.505513 18212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:12:37.506158 18226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.506309 18212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:37.506429 18212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "b7c4133b55be495c9d85df5a0f98812c"
format_stamp: "Formatted at 2026-09-24 05:12:37 on dist-test-slave-mr2f"
server_key: "2688739350be10f86ea15e93db67acbd"
server_key_iv: "fc340b2d2d176cc06bcfe6ca9a2ae097"
server_key_version: "encryptionkey@0"
I20260924 05:12:37.506516 18212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:37.519234 18212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:37.519515 18212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:37.519659 18212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:37.519892 18212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:37.520219 18212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:37.520272 18212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.520308 18212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:37.520334 18212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:37.525736 18212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38039
I20260924 05:12:37.525839 18339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38039 every 8 connection(s)
I20260924 05:12:37.526129 18212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:12:37.528199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 18212
I20260924 05:12:37.528311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:12:37.528513 392 external_mini_cluster.cc:1428] Setting key 0ca259b97a943ad2448b74b9f14d8697
I20260924 05:12:37.531594 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.526853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34581 (local address 127.0.98.62:39641)
0924 05:12:37.526924 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.526926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.527185 (+ 259us) server_negotiation.cc:407] Connection header received
0924 05:12:37.527635 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.527636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.527646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.527701 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:37.528366 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.529038 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.530170 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.530386 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.530467 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.531306 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:37.531308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:37.531308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:37.531313 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:37.531327 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:37.531331 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:37.531387 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.531487 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.531514 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:12:37.531826 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.526862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39641 (local address 127.0.98.3:34581)
0924 05:12:37.527039 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.527089 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:37.527552 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:37.527753 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:37.527756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:37.527764 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:37.528248 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.528252 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.529180 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.529183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.530063 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.530067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.530477 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.530479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.530618 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.531153 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:12:37.531154 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:37.531182 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:12:37.531206 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:37.531387 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:37.531392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:37.531395 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:37.531490 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 05:12:37.531593 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260924 05:12:37.532254 18340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39641
I20260924 05:12:37.532367 18340 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:37.532543 18340 heartbeater.cc:511] Master 127.0.98.62:39641 requested a full tablet report, sending...
I20260924 05:12:37.532902 17890 ts_manager.cc:194] Registered new tserver with Master: b7c4133b55be495c9d85df5a0f98812c (127.0.98.3:38039)
I20260924 05:12:37.533267 17890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34581
I20260924 05:12:37.540299 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:12:37.544651 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.541752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.0.98.62:39641)
0924 05:12:37.541804 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.541823 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.541943 (+ 120us) server_negotiation.cc:407] Connection header received
0924 05:12:37.542019 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.542020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.542035 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.542093 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:37.542262 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.542724 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.543538 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.543725 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.543801 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.544404 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:37.544405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:37.544405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:37.544410 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:37.544422 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:37.544426 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:37.544494 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.544549 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.544569 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:12:37.545989 17890 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:12:37.547544 17890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:12:37.553443 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.549499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40769 (local address 127.0.0.1:54646)
0924 05:12:37.549817 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.549853 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:37.549926 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:37.550133 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:37.550137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:37.550139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:37.550204 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.550206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.551523 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.551524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.552366 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.552369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.553234 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.553235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.553287 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.553297 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 05:12:37.553360 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:12:37.553394 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260924 05:12:37.553507 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.549614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.0.98.2:40769)
0924 05:12:37.549814 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.549818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.549935 (+ 117us) server_negotiation.cc:407] Connection header received
0924 05:12:37.550004 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.550006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.550024 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.550118 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:37.550289 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.551424 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.552453 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.553154 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.553230 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.553317 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.553378 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.553412 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260924 05:12:37.553983 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.549474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38039 (local address 127.0.0.1:46136)
0924 05:12:37.549813 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.549853 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:37.549919 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:37.550239 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:37.550240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:37.550242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:37.550293 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.550295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.551587 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.551588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.552671 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.552673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.553805 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.553805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.553872 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.553875 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:37.553913 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 05:12:37.553940 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":67,"threads_started":1}
I20260924 05:12:37.554231 18143 tablet_service.cc:1467] Processing CreateTablet for tablet 5f67cf0197ed4a4bba2b1a374f64d993 (DEFAULT_TABLE table=test-table [id=6559c00e7c1f4706a98ca41014d9f0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:37.554615 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.549706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41021 (local address 127.0.0.1:48340)
0924 05:12:37.550079 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.550089 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:37.550137 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:37.550431 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:37.550431 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:37.550433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:37.550469 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.550470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.551416 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.551417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.552678 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.552679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.554382 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.554383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.554459 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.554464 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:12:37.554520 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:12:37.554557 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":34,"thread_start_us":68,"threads_started":1}
I20260924 05:12:37.554610 18143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f67cf0197ed4a4bba2b1a374f64d993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:37.554939 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.549824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.0.98.1:41021)
0924 05:12:37.550021 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.550026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.550192 (+ 166us) server_negotiation.cc:407] Connection header received
0924 05:12:37.550234 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.550236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.550249 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.550316 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:37.550540 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.551330 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.553631 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.554293 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.554370 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.554465 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.554541 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.554582 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":97,"threads_started":1}
I20260924 05:12:37.555037 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.549551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.0.98.3:38039)
0924 05:12:37.550058 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.550062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.550098 (+ 36us) server_negotiation.cc:407] Connection header received
0924 05:12:37.550134 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.550136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.550150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.550223 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:37.550348 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.551511 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.552755 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.553446 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.554487 (+ 1041us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.554571 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.554791 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.554900 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":73,"threads_started":1}
I20260924 05:12:37.555523 18012 tablet_service.cc:1467] Processing CreateTablet for tablet 5f67cf0197ed4a4bba2b1a374f64d993 (DEFAULT_TABLE table=test-table [id=6559c00e7c1f4706a98ca41014d9f0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:37.555586 18274 tablet_service.cc:1467] Processing CreateTablet for tablet 5f67cf0197ed4a4bba2b1a374f64d993 (DEFAULT_TABLE table=test-table [id=6559c00e7c1f4706a98ca41014d9f0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:37.555869 18274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f67cf0197ed4a4bba2b1a374f64d993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:37.555881 18012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f67cf0197ed4a4bba2b1a374f64d993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:37.557394 18354 tablet_bootstrap.cc:492] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b: Bootstrap starting.
I20260924 05:12:37.558135 18354 tablet_bootstrap.cc:654] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:37.558435 18354 log.cc:824] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:37.559228 18354 tablet_bootstrap.cc:492] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b: No bootstrap required, opened a new log
I20260924 05:12:37.559320 18354 ts_tablet_manager.cc:1397] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:12:37.561172 18354 raft_consensus.cc:348] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } }
I20260924 05:12:37.561347 18354 raft_consensus.cc:374] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:37.561393 18354 raft_consensus.cc:729] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af458ffb9ad1400d8e660c1f3e19fb3b, State: Initialized, Role: FOLLOWER
I20260924 05:12:37.561517 18354 consensus_queue.cc:260] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } }
I20260924 05:12:37.561774 18354 ts_tablet_manager.cc:1428] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:12:37.561843 18209 heartbeater.cc:503] Master 127.0.98.62:39641 was elected leader, sending a full tablet report...
I20260924 05:12:37.561975 18356 tablet_bootstrap.cc:492] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b: Bootstrap starting.
I20260924 05:12:37.562602 18356 tablet_bootstrap.cc:654] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:37.562922 18357 tablet_bootstrap.cc:492] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: Bootstrap starting.
I20260924 05:12:37.562990 18356 log.cc:824] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:37.563534 18357 tablet_bootstrap.cc:654] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:37.563724 18356 tablet_bootstrap.cc:492] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b: No bootstrap required, opened a new log
I20260924 05:12:37.563794 18356 ts_tablet_manager.cc:1397] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:12:37.563853 18357 log.cc:824] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:37.564587 18357 tablet_bootstrap.cc:492] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: No bootstrap required, opened a new log
I20260924 05:12:37.564654 18357 ts_tablet_manager.cc:1397] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:12:37.566367 18356 raft_consensus.cc:348] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } }
I20260924 05:12:37.566548 18356 raft_consensus.cc:374] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:37.566586 18356 raft_consensus.cc:729] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643368f03a0445869f6b7ee59515684b, State: Initialized, Role: FOLLOWER
I20260924 05:12:37.566701 18356 consensus_queue.cc:260] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } }
I20260924 05:12:37.566985 18356 ts_tablet_manager.cc:1428] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 05:12:37.567023 18078 heartbeater.cc:503] Master 127.0.98.62:39641 was elected leader, sending a full tablet report...
I20260924 05:12:37.567196 18357 raft_consensus.cc:348] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } }
I20260924 05:12:37.567365 18357 raft_consensus.cc:374] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:37.567404 18357 raft_consensus.cc:729] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c4133b55be495c9d85df5a0f98812c, State: Initialized, Role: FOLLOWER
I20260924 05:12:37.567510 18357 consensus_queue.cc:260] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } }
I20260924 05:12:37.567776 18357 ts_tablet_manager.cc:1428] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:12:37.567823 18340 heartbeater.cc:503] Master 127.0.98.62:39641 was elected leader, sending a full tablet report...
I20260924 05:12:37.628371 18358 raft_consensus.cc:482] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:12:37.628489 18358 raft_consensus.cc:504] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } }
I20260924 05:12:37.628892 18358 leader_election.cc:288] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7c4133b55be495c9d85df5a0f98812c (127.0.98.3:38039), 643368f03a0445869f6b7ee59515684b (127.0.98.1:41021)
I20260924 05:12:37.631620 18211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.629046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38039 (local address 127.0.98.2:56631)
0924 05:12:37.629116 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.629130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:37.629193 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:37.629409 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:37.629410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:37.629411 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:37.629481 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.629483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.630101 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.630103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.630871 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.630873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.631370 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.631371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.631442 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.631445 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:37.631515 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:12:37.631555 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 05:12:37.631620 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.629135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56631 (local address 127.0.98.3:38039)
0924 05:12:37.629183 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.629186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.629209 (+ 23us) server_negotiation.cc:407] Connection header received
0924 05:12:37.629290 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.629291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.629304 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.629399 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:37.629561 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.630021 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.630953 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.631284 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.631370 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.631379 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.631523 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.631555 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:12:37.631887 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.629227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42483 (local address 127.0.98.1:41021)
0924 05:12:37.629269 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.629271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.629394 (+ 123us) server_negotiation.cc:407] Connection header received
0924 05:12:37.629452 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.629453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.629465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.629534 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:37.629696 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.630143 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.631114 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.631480 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.631564 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.631574 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.631751 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.631817 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:12:37.631896 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.629159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41021 (local address 127.0.98.2:42483)
0924 05:12:37.629329 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.629337 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:37.629397 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:37.629550 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:37.629551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:37.629553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:37.629625 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.629627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.630249 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.630250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.631026 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:37.631028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.631574 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.631576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:37.631643 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.631645 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:12:37.631728 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:12:37.631791 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":111,"threads_started":1}
I20260924 05:12:37.631986 18294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f67cf0197ed4a4bba2b1a374f64d993" candidate_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7c4133b55be495c9d85df5a0f98812c" is_pre_election: true
I20260924 05:12:37.632119 18294 raft_consensus.cc:2454] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af458ffb9ad1400d8e660c1f3e19fb3b in term 0.
I20260924 05:12:37.632133 18032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f67cf0197ed4a4bba2b1a374f64d993" candidate_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643368f03a0445869f6b7ee59515684b" is_pre_election: true
I20260924 05:12:37.632247 18032 raft_consensus.cc:2454] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af458ffb9ad1400d8e660c1f3e19fb3b in term 0.
I20260924 05:12:37.632356 18098 leader_election.cc:302] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af458ffb9ad1400d8e660c1f3e19fb3b, b7c4133b55be495c9d85df5a0f98812c; no voters:
I20260924 05:12:37.632471 18358 raft_consensus.cc:2781] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:12:37.632539 18358 raft_consensus.cc:482] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:12:37.632563 18358 raft_consensus.cc:3037] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:37.633260 18358 raft_consensus.cc:504] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } }
I20260924 05:12:37.633390 18358 leader_election.cc:288] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [CANDIDATE]: Term 1 election: Requested vote from peers b7c4133b55be495c9d85df5a0f98812c (127.0.98.3:38039), 643368f03a0445869f6b7ee59515684b (127.0.98.1:41021)
I20260924 05:12:37.633569 18294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f67cf0197ed4a4bba2b1a374f64d993" candidate_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7c4133b55be495c9d85df5a0f98812c"
I20260924 05:12:37.633644 18294 raft_consensus.cc:3037] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:37.633623 18032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f67cf0197ed4a4bba2b1a374f64d993" candidate_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643368f03a0445869f6b7ee59515684b"
I20260924 05:12:37.633755 18032 raft_consensus.cc:3037] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:37.634367 18294 raft_consensus.cc:2454] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af458ffb9ad1400d8e660c1f3e19fb3b in term 1.
I20260924 05:12:37.634549 18098 leader_election.cc:302] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af458ffb9ad1400d8e660c1f3e19fb3b, b7c4133b55be495c9d85df5a0f98812c; no voters:
I20260924 05:12:37.634563 18032 raft_consensus.cc:2454] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af458ffb9ad1400d8e660c1f3e19fb3b in term 1.
I20260924 05:12:37.634642 18358 raft_consensus.cc:2781] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:37.634831 18358 raft_consensus.cc:686] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 1 LEADER]: Becoming Leader. State: Replica: af458ffb9ad1400d8e660c1f3e19fb3b, State: Running, Role: LEADER
I20260924 05:12:37.634943 18358 consensus_queue.cc:237] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } }
I20260924 05:12:37.635768 17890 catalog_manager.cc:5495] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b reported cstate change: term changed from 0 to 1, leader changed from <none> to af458ffb9ad1400d8e660c1f3e19fb3b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af458ffb9ad1400d8e660c1f3e19fb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20260924 05:12:37.644655 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.642644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.0.98.2:40769)
0924 05:12:37.642683 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:37.642685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:37.642692 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:12:37.642801 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:37.642802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:37.642820 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:37.642892 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:12:37.642991 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.643395 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.644040 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:37.644206 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:37.644252 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:37.644362 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:12:37.644445 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:12:37.644501 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:37.644529 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 05:12:37.644542 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:12:37.646217 18032 raft_consensus.cc:1261] T 5f67cf0197ed4a4bba2b1a374f64d993 P 643368f03a0445869f6b7ee59515684b [term 1 FOLLOWER]: Refusing update from remote peer af458ffb9ad1400d8e660c1f3e19fb3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:37.646518 18294 raft_consensus.cc:1261] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [term 1 FOLLOWER]: Refusing update from remote peer af458ffb9ad1400d8e660c1f3e19fb3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:37.646508 18358 consensus_queue.cc:1035] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "643368f03a0445869f6b7ee59515684b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:37.646785 18358 consensus_queue.cc:1035] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7c4133b55be495c9d85df5a0f98812c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:37.649159 18368 mvcc.cc:205] Tried to move back new op lower bound from 7332768799316660224 to 7332768799273443328. Current Snapshot: MvccSnapshot[applied={T|T < 7332768799316660224}]
I20260924 05:12:37.649209 18369 mvcc.cc:205] Tried to move back new op lower bound from 7332768799316660224 to 7332768799273443328. Current Snapshot: MvccSnapshot[applied={T|T < 7332768799316660224}]
I20260924 05:12:37.651355 17889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:37.651446 17889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:37.652212 17889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d208c8dfd9940d7b1f4973305ecfb3c: Sending DeleteTablet for 3 replicas of tablet 5f67cf0197ed4a4bba2b1a374f64d993
I20260924 05:12:37.652551 18143 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f67cf0197ed4a4bba2b1a374f64d993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:37 UTC) from {username='slave'} at 127.0.0.1:54646
I20260924 05:12:37.652551 18012 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f67cf0197ed4a4bba2b1a374f64d993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:37 UTC) from {username='slave'} at 127.0.0.1:48340
I20260924 05:12:37.652570 18274 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f67cf0197ed4a4bba2b1a374f64d993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:37 UTC) from {username='slave'} at 127.0.0.1:46136
I20260924 05:12:37.652760 18379 tablet_replica.cc:330] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: stopping tablet replica
I20260924 05:12:37.652859 18379 raft_consensus.cc:2229] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:12:37.652952 18379 raft_consensus.cc:2258] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:37.653007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 17950
I20260924 05:12:37.653942 18379 ts_tablet_manager.cc:1905] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:37.655520 18379 ts_tablet_manager.cc:1918] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:37.655597 18379 log.cc:1196] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/5f67cf0197ed4a4bba2b1a374f64d993
I20260924 05:12:37.655876 18379 ts_tablet_manager.cc:1939] T 5f67cf0197ed4a4bba2b1a374f64d993 P b7c4133b55be495c9d85df5a0f98812c: Deleting consensus metadata
I20260924 05:12:37.656368 17876 catalog_manager.cc:4841] TS b7c4133b55be495c9d85df5a0f98812c (127.0.98.3:38039): tablet 5f67cf0197ed4a4bba2b1a374f64d993 (table test-table [id=6559c00e7c1f4706a98ca41014d9f0c2]) successfully deleted
W20260924 05:12:37.659824 17876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41021 (error 108)
I20260924 05:12:37.660056 18378 tablet_replica.cc:330] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b: stopping tablet replica
I20260924 05:12:37.660159 18378 raft_consensus.cc:2229] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:12:37.660277 18378 raft_consensus.cc:2258] T 5f67cf0197ed4a4bba2b1a374f64d993 P af458ffb9ad1400d8e660c1f3e19fb3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:37.660452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 18081
I20260924 05:12:37.661896 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.661764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41021 (local address 127.0.0.1:48346)
0924 05:12:37.661821 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.661870 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 05:12:37.662019 17876 catalog_manager.cc:4584] TS 643368f03a0445869f6b7ee59515684b (127.0.98.1:41021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f67cf0197ed4a4bba2b1a374f64d993: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41021: connect: Connection refused (error 111)
I20260924 05:12:37.665755 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:37.665632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40769 (local address 127.0.0.1:54668)
0924 05:12:37.665697 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:37.665728 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:12:37.665750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 18212
I20260924 05:12:37.670184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 17859
2026-09-24T05:12:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (674 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:37 dist-test-slave-mr2f krb5kdc[18383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:37 dist-test-slave-mr2f krb5kdc[18383](info): set up 2 sockets
Sep 24 05:12:37 dist-test-slave-mr2f krb5kdc[18383](info): commencing operation
krb5kdc: starting...
W20260924 05:12:38.694433 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:38 dist-test-slave-mr2f krb5kdc[18383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:12:38.807488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42031
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:38.886448 18396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:38.886672 18396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:38.886708 18396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:38.887866 18396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:38.887938 18396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:38.887966 18396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:38.887996 18396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:38.888036 18396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:38.889565 18396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:38.889842 18396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:38.890059 18396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:12:38.894872 18396 server_base.cc:1034] running on GCE node
W20260924 05:12:38.894851 18404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:38.894841 18402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:38.894847 18401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:38.895257 18396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:38.895540 18396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:38.896682 18396 hybrid_clock.cc:648] HybridClock initialized: now 1790226758896662 us; error 41 us; skew 500 ppm
Sep 24 05:12:38 dist-test-slave-mr2f krb5kdc[18383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:38.899000 18396 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:12:38.899502 18396 webserver.cc:460] Webserver started at http://127.0.98.62:41385/ using document root <none> and password file <none>
I20260924 05:12:38.899734 18396 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:38.899776 18396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:38.899919 18396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:38.900965 18396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "e49cb809ca0543fe93a789c4c20550f2"
format_stamp: "Formatted at 2026-09-24 05:12:38 on dist-test-slave-mr2f"
server_key: "4a3e7d298b0b577584fdf23dbc5b857b"
server_key_iv: "ae29547b393874b26bed6c0082c66527"
server_key_version: "encryptionkey@0"
I20260924 05:12:38.901328 18396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "e49cb809ca0543fe93a789c4c20550f2"
format_stamp: "Formatted at 2026-09-24 05:12:38 on dist-test-slave-mr2f"
server_key: "4a3e7d298b0b577584fdf23dbc5b857b"
server_key_iv: "ae29547b393874b26bed6c0082c66527"
server_key_version: "encryptionkey@0"
I20260924 05:12:38.902599 18396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:12:38.903355 18411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:38.903518 18396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:38.903613 18396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "e49cb809ca0543fe93a789c4c20550f2"
format_stamp: "Formatted at 2026-09-24 05:12:38 on dist-test-slave-mr2f"
server_key: "4a3e7d298b0b577584fdf23dbc5b857b"
server_key_iv: "ae29547b393874b26bed6c0082c66527"
server_key_version: "encryptionkey@0"
I20260924 05:12:38.903697 18396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:38.926831 18396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:38.928733 18396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:38.928908 18396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:38.932821 18396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42031
I20260924 05:12:38.932858 18463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42031 every 8 connection(s)
I20260924 05:12:38.933346 18396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:38.933992 18464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:38.936292 18464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2: Bootstrap starting.
I20260924 05:12:38.936851 18464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:38.937142 18464 log.cc:824] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:38.937772 18464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2: No bootstrap required, opened a new log
I20260924 05:12:38.939287 18464 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49cb809ca0543fe93a789c4c20550f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42031 } }
I20260924 05:12:38.939425 18464 raft_consensus.cc:374] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:38.939482 18464 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49cb809ca0543fe93a789c4c20550f2, State: Initialized, Role: FOLLOWER
I20260924 05:12:38.939648 18464 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49cb809ca0543fe93a789c4c20550f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42031 } }
I20260924 05:12:38.939746 18464 raft_consensus.cc:388] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:38.939796 18464 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:38.939845 18464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:38.940712 18464 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49cb809ca0543fe93a789c4c20550f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42031 } }
I20260924 05:12:38.940820 18464 leader_election.cc:302] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49cb809ca0543fe93a789c4c20550f2; no voters:
I20260924 05:12:38.941023 18464 leader_election.cc:288] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:38.941049 18467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:38.941195 18467 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [term 1 LEADER]: Becoming Leader. State: Replica: e49cb809ca0543fe93a789c4c20550f2, State: Running, Role: LEADER
I20260924 05:12:38.941274 18464 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:38.941318 18467 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49cb809ca0543fe93a789c4c20550f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42031 } }
I20260924 05:12:38.941679 18469 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49cb809ca0543fe93a789c4c20550f2. Latest consensus state: current_term: 1 leader_uuid: "e49cb809ca0543fe93a789c4c20550f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49cb809ca0543fe93a789c4c20550f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42031 } } }
I20260924 05:12:38.941669 18468 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49cb809ca0543fe93a789c4c20550f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49cb809ca0543fe93a789c4c20550f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42031 } } }
I20260924 05:12:38.941849 18468 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:38.942041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 18396
I20260924 05:12:38.942030 18469 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:38.942162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:38.942355 392 external_mini_cluster.cc:1428] Setting key 60145703a1217d5faed7d8179671af51
I20260924 05:12:38.942873 18476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:38.943410 18476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:38.945097 18476 catalog_manager.cc:1347] Generated new cluster ID: 3308f5ed28e54f81b942b5b1809aba91
I20260924 05:12:38.945147 18476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:12:38 dist-test-slave-mr2f krb5kdc[18383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:38.954559 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:38.942914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.0.98.62:42031)
0924 05:12:38.943080 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:38.943089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:38.943553 (+ 464us) server_negotiation.cc:407] Connection header received
0924 05:12:38.943761 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:38.943769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:38.943824 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:38.943936 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:38.944893 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:38.946264 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:38.947389 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:38.948018 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:38.948113 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:38.950867 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:38.950871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:38.950871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:38.950883 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:38.952852 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:38.953236 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:38.953239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:38.953239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:38.953370 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:38.953587 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:38.953588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:38.953588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:38.953897 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:38.954058 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:38.954180 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 05:12:38.954316 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260924 05:12:38.960991 18476 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:38.961475 18476 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:38.970182 18476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2: Generated new TSK 0
I20260924 05:12:38.970362 18476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:12:39.000927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42031
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:39.079768 18490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:39.079955 18490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:39.079979 18490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:39.081100 18490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:39.081141 18490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:39.081167 18490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:12:39.082583 18490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:39.082831 18490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:39.083057 18490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:39.087761 18496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.087762 18495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.087762 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:39.087986 18490 server_base.cc:1034] running on GCE node
I20260924 05:12:39.088258 18490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:39.088485 18490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:39.089658 18490 hybrid_clock.cc:648] HybridClock initialized: now 1790226759089649 us; error 40 us; skew 500 ppm
Sep 24 05:12:39 dist-test-slave-mr2f krb5kdc[18383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:39.092232 18490 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:12:39.092636 18490 webserver.cc:460] Webserver started at http://127.0.98.1:45027/ using document root <none> and password file <none>
I20260924 05:12:39.092857 18490 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:39.092932 18490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:39.093080 18490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:39.094097 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "f66b0ade3cdc4c9bb966439857e061c1"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "cc2ecf47c0d8c28c5bde660be5103ce1"
server_key_iv: "6ad87bc0dd160812925b38e9a29d2ca0"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.094455 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f66b0ade3cdc4c9bb966439857e061c1"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "cc2ecf47c0d8c28c5bde660be5103ce1"
server_key_iv: "6ad87bc0dd160812925b38e9a29d2ca0"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.095849 18490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:12:39.096541 18505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.096702 18490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.096813 18490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "f66b0ade3cdc4c9bb966439857e061c1"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "cc2ecf47c0d8c28c5bde660be5103ce1"
server_key_iv: "6ad87bc0dd160812925b38e9a29d2ca0"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.096886 18490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:39.112207 18490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:39.113971 18490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:39.114130 18490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:39.114360 18490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:39.114658 18490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:39.114712 18490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.114754 18490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:39.114805 18490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.120419 18490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32961
I20260924 05:12:39.120492 18618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32961 every 8 connection(s)
I20260924 05:12:39.120796 18490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:12:39.125298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 18490
I20260924 05:12:39.125408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:12:39.125614 392 external_mini_cluster.cc:1428] Setting key e604e56deaf2e8a671f44c21cf3a16cb
Sep 24 05:12:39 dist-test-slave-mr2f krb5kdc[18383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:39.130376 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.121492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50071 (local address 127.0.98.62:42031)
0924 05:12:39.121562 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.121564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.121767 (+ 203us) server_negotiation.cc:407] Connection header received
0924 05:12:39.122675 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.122676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.122689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.122745 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:39.123399 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.123885 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.124797 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.124979 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.125027 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.127488 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:39.127489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:39.127490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:39.127495 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:39.128977 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:39.129348 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:39.129350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:39.129350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:39.129432 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:39.129693 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:39.129693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:39.129694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:39.129808 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:39.129884 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.130173 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.130272 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 05:12:39.130509 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.121468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42031 (local address 127.0.98.1:50071)
0924 05:12:39.121639 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.121681 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:39.122586 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:39.122893 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:39.122897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:39.123145 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:39.123323 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.123327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.123960 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.123961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.124725 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.124727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.125047 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.125048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.125138 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.125625 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:12:39.125633 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:39.127402 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:39.129046 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:39.129052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:39.129053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:39.129256 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:39.129509 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:39.129510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:39.129510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:39.129619 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:39.129876 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:39.129882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:39.130018 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 05:12:39.130150 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 05:12:39.130296 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260924 05:12:39.130983 18619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42031
I20260924 05:12:39.131090 18619 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:39.131286 18619 heartbeater.cc:511] Master 127.0.98.62:42031 requested a full tablet report, sending...
I20260924 05:12:39.131779 18428 ts_manager.cc:194] Registered new tserver with Master: f66b0ade3cdc4c9bb966439857e061c1 (127.0.98.1:32961)
I20260924 05:12:39.132404 18428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50071
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:12:39.149622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42031
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:39.228363 18624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:39.228562 18624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:39.228585 18624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:39.229727 18624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:39.229772 18624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:39.229830 18624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:12:39.231303 18624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:39.231552 18624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:39.231782 18624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:39.236789 18632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:39.236848 18624 server_base.cc:1034] running on GCE node
W20260924 05:12:39.236897 18630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.237035 18629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:39.237311 18624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:39.237552 18624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:39.238719 18624 hybrid_clock.cc:648] HybridClock initialized: now 1790226759238701 us; error 30 us; skew 500 ppm
Sep 24 05:12:39 dist-test-slave-mr2f krb5kdc[18383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:39.247529 18624 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:12:39.247984 18624 webserver.cc:460] Webserver started at http://127.0.98.2:44615/ using document root <none> and password file <none>
I20260924 05:12:39.248209 18624 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:39.248272 18624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:39.248412 18624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:39.249380 18624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "1de790d656bb492f9b25895606ee81f4"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "50fe9acca238b1b9b07c188f8e6a19d5"
server_key_iv: "be71ac8cd4fe77adb2f46c06d203dba2"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.249737 18624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1de790d656bb492f9b25895606ee81f4"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "50fe9acca238b1b9b07c188f8e6a19d5"
server_key_iv: "be71ac8cd4fe77adb2f46c06d203dba2"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.251093 18624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:12:39.251776 18639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.251966 18624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:12:39.252064 18624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "1de790d656bb492f9b25895606ee81f4"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "50fe9acca238b1b9b07c188f8e6a19d5"
server_key_iv: "be71ac8cd4fe77adb2f46c06d203dba2"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.252134 18624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:39.260026 18624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:39.261659 18624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:39.261812 18624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:39.262034 18624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:39.262313 18624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:39.262367 18624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.262399 18624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:39.262428 18624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.268050 18624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41609
I20260924 05:12:39.268066 18752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41609 every 8 connection(s)
I20260924 05:12:39.268366 18624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:12:39.273908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 18624
I20260924 05:12:39.274039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:12:39.274220 392 external_mini_cluster.cc:1428] Setting key 7ad4b0e688129b939a5632a5a44033ff
Sep 24 05:12:39 dist-test-slave-mr2f krb5kdc[18383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:39.277663 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.269001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42031 (local address 127.0.98.2:60947)
0924 05:12:39.269157 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.269189 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:39.270193 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:39.270410 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:39.270413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:39.270643 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:39.270894 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.270899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.271528 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.271528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.272308 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.272309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.272631 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.272632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.272726 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.273241 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:12:39.273249 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:39.274688 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:39.276299 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:39.276306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:39.276308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:39.276618 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:39.276804 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:39.276805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:39.276806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:39.276959 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:39.277169 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:39.277174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:39.277334 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 05:12:39.277418 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:12:39.277527 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260924 05:12:39.277691 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.269045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60947 (local address 127.0.98.62:42031)
0924 05:12:39.269118 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.269121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.269281 (+ 160us) server_negotiation.cc:407] Connection header received
0924 05:12:39.270280 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.270282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.270292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.270344 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:39.270970 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.271454 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.272379 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.272564 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.272614 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.274801 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:39.274802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:39.274803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:39.274809 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:39.276236 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:39.276692 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:39.276693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:39.276693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:39.276740 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:39.277020 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:39.277021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:39.277021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:39.277107 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:39.277153 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.277468 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.277556 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 05:12:39.278191 18753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42031
I20260924 05:12:39.278294 18753 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:39.278450 18753 heartbeater.cc:511] Master 127.0.98.62:42031 requested a full tablet report, sending...
I20260924 05:12:39.278820 18428 ts_manager.cc:194] Registered new tserver with Master: 1de790d656bb492f9b25895606ee81f4 (127.0.98.2:41609)
I20260924 05:12:39.279304 18428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60947
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:12:39.297088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42031
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:39.373927 18758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:39.374116 18758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:39.374140 18758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:39.375298 18758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:39.375339 18758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:39.375420 18758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:12:39.376884 18758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:39.377123 18758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:39.377341 18758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:12:39.382251 18758 server_base.cc:1034] running on GCE node
W20260924 05:12:39.382162 18766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.382162 18763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.382191 18764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:39.382587 18758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:39.382854 18758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:39.384019 18758 hybrid_clock.cc:648] HybridClock initialized: now 1790226759384004 us; error 28 us; skew 500 ppm
Sep 24 05:12:39 dist-test-slave-mr2f krb5kdc[18383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:39.386304 18758 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:12:39.386848 18758 webserver.cc:460] Webserver started at http://127.0.98.3:41493/ using document root <none> and password file <none>
I20260924 05:12:39.387111 18758 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:39.387172 18758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:39.387310 18758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:39.388545 18758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "43e963b31e5d41678d956b057c338849"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "24d4aed3e8337b7167c715c5fdb1c7fd"
server_key_iv: "401c6a4b9c16d8548848b0a086a5ec2a"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.388896 18758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "43e963b31e5d41678d956b057c338849"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "24d4aed3e8337b7167c715c5fdb1c7fd"
server_key_iv: "401c6a4b9c16d8548848b0a086a5ec2a"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.390149 18758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:12:39.390877 18773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.391022 18758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:12:39.391100 18758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "43e963b31e5d41678d956b057c338849"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "24d4aed3e8337b7167c715c5fdb1c7fd"
server_key_iv: "401c6a4b9c16d8548848b0a086a5ec2a"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.391148 18758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:39.405647 18758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:39.408128 18758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:39.408234 18758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:39.408406 18758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:39.408710 18758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:39.408742 18758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.408787 18758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:39.408814 18758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.414367 18758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45915
I20260924 05:12:39.414435 18886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45915 every 8 connection(s)
I20260924 05:12:39.414750 18758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:12:39 dist-test-slave-mr2f krb5kdc[18383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:39.421368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 18758
I20260924 05:12:39.421456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:12:39.421684 392 external_mini_cluster.cc:1428] Setting key 0efe84f9c219515b4ded3fefd79bedd7
I20260924 05:12:39.423695 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.415459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39693 (local address 127.0.98.62:42031)
0924 05:12:39.415516 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.415518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.415727 (+ 209us) server_negotiation.cc:407] Connection header received
0924 05:12:39.416599 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.416600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.416611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.416661 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:39.417202 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.417672 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.418567 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.418735 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.418858 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.420738 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:39.420739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:39.420739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:39.420745 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:39.422282 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:39.422697 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:39.422699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:39.422699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:39.422796 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:39.423038 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:39.423039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:39.423040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:39.423167 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:39.423241 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.423517 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.423595 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:12:39.423869 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.415437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42031 (local address 127.0.98.3:39693)
0924 05:12:39.415586 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.415617 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:39.416512 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:39.416713 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:39.416716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:39.416963 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:39.417136 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.417140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.417743 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.417744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.418495 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.418497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.418862 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.418863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.418998 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.419383 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:12:39.419390 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:39.420658 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:39.422388 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:39.422394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:39.422396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:39.422607 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:39.422867 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:39.422868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:39.422868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:39.422968 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:39.423233 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:39.423239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:39.423390 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 05:12:39.423494 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 05:12:39.423652 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 05:12:39.424276 18887 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42031
I20260924 05:12:39.424372 18887 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:39.424505 18887 heartbeater.cc:511] Master 127.0.98.62:42031 requested a full tablet report, sending...
I20260924 05:12:39.424853 18428 ts_manager.cc:194] Registered new tserver with Master: 43e963b31e5d41678d956b057c338849 (127.0.98.3:45915)
I20260924 05:12:39.425172 18428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39693
I20260924 05:12:39.433416 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:12:39.440258 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.434648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.0.98.62:42031)
0924 05:12:39.434704 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.434705 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.434862 (+ 157us) server_negotiation.cc:407] Connection header received
0924 05:12:39.434919 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.434919 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.434927 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.434970 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:39.435457 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.435930 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.436609 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.436780 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.436826 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.438075 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:39.438076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:39.438076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:39.438081 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:39.439253 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:39.439604 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:39.439604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:39.439605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:39.439662 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:39.439833 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:39.439833 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:39.439833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:39.439943 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:39.440018 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.440110 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.440154 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:12:39.441699 18428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:12:39.443625 18428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:12:39.449746 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.445586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32961 (local address 127.0.0.1:41400)
0924 05:12:39.445911 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.445963 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:39.446025 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:39.446550 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:39.446551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:39.446553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:39.446598 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.446599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.447812 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.447813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.448685 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.448688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.449535 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.449535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.449598 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.449609 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 05:12:39.449663 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 05:12:39.449697 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":82,"threads_started":1}
I20260924 05:12:39.449792 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.445713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.98.1:32961)
0924 05:12:39.446390 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.446394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.446403 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:12:39.446440 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.446442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.446456 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.446665 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:39.446740 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.447725 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.448788 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.449455 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.449531 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.449587 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.449678 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.449718 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":106,"threads_started":1}
I20260924 05:12:39.450101 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.445618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41609 (local address 127.0.0.1:32862)
0924 05:12:39.445911 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.445963 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:39.446024 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:39.446291 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:39.446292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:39.446294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:39.446454 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.446456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.447399 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.447400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.448663 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.448666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.449854 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.449855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.449938 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.449942 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:12:39.449998 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:12:39.450037 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":52,"threads_started":1}
I20260924 05:12:39.450124 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.445741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.0.98.2:41609)
0924 05:12:39.446161 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.446164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.446172 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:12:39.446199 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.446201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.446212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.446294 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:39.446534 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.447299 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.448749 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.449455 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.449531 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.449587 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.450013 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.450054 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":68,"threads_started":1}
I20260924 05:12:39.450817 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.445816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45915 (local address 127.0.0.1:33080)
0924 05:12:39.445911 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.445968 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:39.446027 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:39.446243 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:39.446246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:39.446248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:39.446317 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.446319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.447600 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.447601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.448663 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.448666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.450453 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.450454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.450511 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.450515 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:12:39.450687 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 05:12:39.450731 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":3,"thread_start_us":42,"threads_started":1}
I20260924 05:12:39.450730 18687 tablet_service.cc:1467] Processing CreateTablet for tablet 92bac577604b4fcdb8e6a0291183a34c (DEFAULT_TABLE table=test-table [id=93f96bb4d1e341b2a2788bf4dd9b7d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:39.450728 18553 tablet_service.cc:1467] Processing CreateTablet for tablet 92bac577604b4fcdb8e6a0291183a34c (DEFAULT_TABLE table=test-table [id=93f96bb4d1e341b2a2788bf4dd9b7d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:39.451082 18687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bac577604b4fcdb8e6a0291183a34c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:39.451112 18553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bac577604b4fcdb8e6a0291183a34c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:39.451179 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.445913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.0.98.3:45915)
0924 05:12:39.446086 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.446090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.446099 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:12:39.446132 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.446134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.446148 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.446240 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:39.446402 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.447503 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.449623 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.450289 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.450368 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.450420 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.450709 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.450792 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260924 05:12:39.451818 18821 tablet_service.cc:1467] Processing CreateTablet for tablet 92bac577604b4fcdb8e6a0291183a34c (DEFAULT_TABLE table=test-table [id=93f96bb4d1e341b2a2788bf4dd9b7d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:39.452102 18821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bac577604b4fcdb8e6a0291183a34c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:39.453680 18901 tablet_bootstrap.cc:492] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1: Bootstrap starting.
I20260924 05:12:39.454268 18902 tablet_bootstrap.cc:492] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: Bootstrap starting.
I20260924 05:12:39.454418 18901 tablet_bootstrap.cc:654] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:39.455003 18902 tablet_bootstrap.cc:654] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:39.455014 18901 log.cc:824] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:39.455215 18903 tablet_bootstrap.cc:492] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: Bootstrap starting.
I20260924 05:12:39.455681 18902 log.cc:824] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:39.455811 18901 tablet_bootstrap.cc:492] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1: No bootstrap required, opened a new log
I20260924 05:12:39.455881 18901 ts_tablet_manager.cc:1397] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:12:39.455991 18903 tablet_bootstrap.cc:654] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:39.456455 18902 tablet_bootstrap.cc:492] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: No bootstrap required, opened a new log
I20260924 05:12:39.456521 18902 ts_tablet_manager.cc:1397] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:12:39.456568 18903 log.cc:824] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:39.457310 18903 tablet_bootstrap.cc:492] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: No bootstrap required, opened a new log
I20260924 05:12:39.457365 18903 ts_tablet_manager.cc:1397] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:12:39.457450 18901 raft_consensus.cc:348] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } }
I20260924 05:12:39.457571 18901 raft_consensus.cc:374] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:39.457590 18901 raft_consensus.cc:729] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66b0ade3cdc4c9bb966439857e061c1, State: Initialized, Role: FOLLOWER
I20260924 05:12:39.457701 18901 consensus_queue.cc:260] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } }
I20260924 05:12:39.457967 18619 heartbeater.cc:503] Master 127.0.98.62:42031 was elected leader, sending a full tablet report...
I20260924 05:12:39.457968 18901 ts_tablet_manager.cc:1428] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:12:39.458446 18902 raft_consensus.cc:348] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } }
I20260924 05:12:39.458609 18902 raft_consensus.cc:374] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:39.458643 18902 raft_consensus.cc:729] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de790d656bb492f9b25895606ee81f4, State: Initialized, Role: FOLLOWER
I20260924 05:12:39.458750 18902 consensus_queue.cc:260] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } }
I20260924 05:12:39.459368 18902 ts_tablet_manager.cc:1428] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:12:39.459291 18903 raft_consensus.cc:348] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } }
I20260924 05:12:39.459375 18753 heartbeater.cc:503] Master 127.0.98.62:42031 was elected leader, sending a full tablet report...
I20260924 05:12:39.459438 18903 raft_consensus.cc:374] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:39.459483 18903 raft_consensus.cc:729] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e963b31e5d41678d956b057c338849, State: Initialized, Role: FOLLOWER
I20260924 05:12:39.459579 18903 consensus_queue.cc:260] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } }
I20260924 05:12:39.459851 18903 ts_tablet_manager.cc:1428] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:12:39.459867 18887 heartbeater.cc:503] Master 127.0.98.62:42031 was elected leader, sending a full tablet report...
I20260924 05:12:39.516542 18908 raft_consensus.cc:482] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:12:39.516659 18908 raft_consensus.cc:504] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } }
I20260924 05:12:39.516957 18908 leader_election.cc:288] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66b0ade3cdc4c9bb966439857e061c1 (127.0.98.1:32961), 43e963b31e5d41678d956b057c338849 (127.0.98.3:45915)
W20260924 05:12:39.518913 18754 tablet.cc:2369] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:12:39.520051 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.517241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48359 (local address 127.0.98.1:32961)
0924 05:12:39.517306 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.517308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.517319 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:12:39.517418 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.517420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.517433 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.517494 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:39.517672 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.518201 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.519331 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.519684 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.519740 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.519747 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.519929 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.519977 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:12:39.520056 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.517179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32961 (local address 127.0.98.2:48359)
0924 05:12:39.517243 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.517257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:39.517316 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:39.517505 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:39.517506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:39.517509 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:39.517582 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.517584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.518273 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.518274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.519243 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.519245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.519761 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.519762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.519853 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.519856 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:39.519911 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 05:12:39.519940 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":17}
I20260924 05:12:39.520102 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.517420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51675 (local address 127.0.98.3:45915)
0924 05:12:39.517468 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.517471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.517505 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:12:39.517584 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.517585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.517594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.517651 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:39.517811 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.518268 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.519367 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.519739 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.519807 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.519816 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.519989 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.520032 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:12:39.520129 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.517285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45915 (local address 127.0.98.2:51675)
0924 05:12:39.517456 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.517465 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:39.517511 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:39.517672 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:39.517672 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:39.517674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:39.517742 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.517743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.518355 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.518356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.519251 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.519253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.519823 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.519824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.519880 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.519908 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 05:12:39.519966 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:12:39.520000 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
I20260924 05:12:39.520427 18841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bac577604b4fcdb8e6a0291183a34c" candidate_uuid: "1de790d656bb492f9b25895606ee81f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e963b31e5d41678d956b057c338849" is_pre_election: true
I20260924 05:12:39.520421 18573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bac577604b4fcdb8e6a0291183a34c" candidate_uuid: "1de790d656bb492f9b25895606ee81f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66b0ade3cdc4c9bb966439857e061c1" is_pre_election: true
I20260924 05:12:39.520587 18841 raft_consensus.cc:2454] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1de790d656bb492f9b25895606ee81f4 in term 0.
I20260924 05:12:39.520601 18573 raft_consensus.cc:2454] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1de790d656bb492f9b25895606ee81f4 in term 0.
I20260924 05:12:39.520823 18642 leader_election.cc:302] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1de790d656bb492f9b25895606ee81f4, 43e963b31e5d41678d956b057c338849; no voters:
I20260924 05:12:39.520948 18908 raft_consensus.cc:2781] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:12:39.521046 18908 raft_consensus.cc:482] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:12:39.521083 18908 raft_consensus.cc:3037] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:39.521777 18908 raft_consensus.cc:504] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } }
I20260924 05:12:39.521919 18908 leader_election.cc:288] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [CANDIDATE]: Term 1 election: Requested vote from peers f66b0ade3cdc4c9bb966439857e061c1 (127.0.98.1:32961), 43e963b31e5d41678d956b057c338849 (127.0.98.3:45915)
I20260924 05:12:39.522122 18573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bac577604b4fcdb8e6a0291183a34c" candidate_uuid: "1de790d656bb492f9b25895606ee81f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66b0ade3cdc4c9bb966439857e061c1"
I20260924 05:12:39.522123 18841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bac577604b4fcdb8e6a0291183a34c" candidate_uuid: "1de790d656bb492f9b25895606ee81f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e963b31e5d41678d956b057c338849"
I20260924 05:12:39.522197 18841 raft_consensus.cc:3037] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:39.522198 18573 raft_consensus.cc:3037] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:39.523056 18573 raft_consensus.cc:2454] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1de790d656bb492f9b25895606ee81f4 in term 1.
I20260924 05:12:39.523056 18841 raft_consensus.cc:2454] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1de790d656bb492f9b25895606ee81f4 in term 1.
I20260924 05:12:39.523236 18642 leader_election.cc:302] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1de790d656bb492f9b25895606ee81f4, 43e963b31e5d41678d956b057c338849; no voters:
I20260924 05:12:39.523321 18908 raft_consensus.cc:2781] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:39.523458 18908 raft_consensus.cc:686] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 1 LEADER]: Becoming Leader. State: Replica: 1de790d656bb492f9b25895606ee81f4, State: Running, Role: LEADER
I20260924 05:12:39.523554 18908 consensus_queue.cc:237] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } }
I20260924 05:12:39.524323 18428 catalog_manager.cc:5495] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1de790d656bb492f9b25895606ee81f4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1de790d656bb492f9b25895606ee81f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1de790d656bb492f9b25895606ee81f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41609 } health_report { overall_health: HEALTHY } } }
I20260924 05:12:39.546255 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.544236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.0.98.2:41609)
0924 05:12:39.544284 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.544286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.544293 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:12:39.544350 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.544351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.544366 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.544424 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:12:39.544519 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.544945 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.545607 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.545774 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.545821 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.545936 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:12:39.546013 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:12:39.546078 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.546120 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.546129 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:12:39.547948 18841 raft_consensus.cc:1261] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [term 1 FOLLOWER]: Refusing update from remote peer 1de790d656bb492f9b25895606ee81f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:39.547948 18573 raft_consensus.cc:1261] T 92bac577604b4fcdb8e6a0291183a34c P f66b0ade3cdc4c9bb966439857e061c1 [term 1 FOLLOWER]: Refusing update from remote peer 1de790d656bb492f9b25895606ee81f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:39.548270 18908 consensus_queue.cc:1035] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66b0ade3cdc4c9bb966439857e061c1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:39.548411 18911 consensus_queue.cc:1035] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e963b31e5d41678d956b057c338849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:39.553377 18427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:39.553470 18427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:39.554199 18427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e49cb809ca0543fe93a789c4c20550f2: Sending DeleteTablet for 3 replicas of tablet 92bac577604b4fcdb8e6a0291183a34c
I20260924 05:12:39.554533 18687 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bac577604b4fcdb8e6a0291183a34c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32862
I20260924 05:12:39.554559 18553 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bac577604b4fcdb8e6a0291183a34c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41400
I20260924 05:12:39.554613 18821 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bac577604b4fcdb8e6a0291183a34c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33080
I20260924 05:12:39.554819 18926 tablet_replica.cc:330] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: stopping tablet replica
I20260924 05:12:39.554932 18926 raft_consensus.cc:2229] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:12:39.554958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 18490
I20260924 05:12:39.554827 18924 tablet_replica.cc:330] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: stopping tablet replica
I20260924 05:12:39.555038 18926 raft_consensus.cc:2258] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:39.555855 18924 raft_consensus.cc:2229] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:12:39.555964 18924 raft_consensus.cc:2258] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:39.556365 18924 ts_tablet_manager.cc:1905] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:39.557055 18926 ts_tablet_manager.cc:1905] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:39.557945 18924 ts_tablet_manager.cc:1918] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:39.558017 18924 log.cc:1196] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/92bac577604b4fcdb8e6a0291183a34c
I20260924 05:12:39.558286 18924 ts_tablet_manager.cc:1939] T 92bac577604b4fcdb8e6a0291183a34c P 1de790d656bb492f9b25895606ee81f4: Deleting consensus metadata
I20260924 05:12:39.558369 18926 ts_tablet_manager.cc:1918] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:39.558434 18926 log.cc:1196] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/92bac577604b4fcdb8e6a0291183a34c
I20260924 05:12:39.558703 18413 catalog_manager.cc:4841] TS 1de790d656bb492f9b25895606ee81f4 (127.0.98.2:41609): tablet 92bac577604b4fcdb8e6a0291183a34c (table test-table [id=93f96bb4d1e341b2a2788bf4dd9b7d49]) successfully deleted
I20260924 05:12:39.558734 18926 ts_tablet_manager.cc:1939] T 92bac577604b4fcdb8e6a0291183a34c P 43e963b31e5d41678d956b057c338849: Deleting consensus metadata
I20260924 05:12:39.559139 18414 catalog_manager.cc:4841] TS 43e963b31e5d41678d956b057c338849 (127.0.98.3:45915): tablet 92bac577604b4fcdb8e6a0291183a34c (table test-table [id=93f96bb4d1e341b2a2788bf4dd9b7d49]) successfully deleted
W20260924 05:12:39.565722 18414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32961 (error 108)
I20260924 05:12:39.565939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 18624
I20260924 05:12:39.566543 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.566058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32961 (local address 127.0.0.1:41412)
0924 05:12:39.566471 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.566507 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
W20260924 05:12:39.566712 18414 catalog_manager.cc:4584] TS f66b0ade3cdc4c9bb966439857e061c1 (127.0.98.1:32961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92bac577604b4fcdb8e6a0291183a34c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32961: connect: Connection refused (error 111)
I20260924 05:12:39.570613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 18758
I20260924 05:12:39.575100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 18396
2026-09-24T05:12:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1905 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T05:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:39Z Disabled control of system clock
I20260924 05:12:39.601838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:12:39.678812 18931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:39.679060 18931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:39.679093 18931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:39.680207 18931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:39.680269 18931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:39.680299 18931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:39.680320 18931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:39.680358 18931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:39.681643 18931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:39.681905 18931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:39.682173 18931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:39.686707 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.686717 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.686754 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:39.686959 18931 server_base.cc:1034] running on GCE node
I20260924 05:12:39.687184 18931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:39.687497 18931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:39.688656 18931 hybrid_clock.cc:648] HybridClock initialized: now 1790226759688639 us; error 38 us; skew 500 ppm
I20260924 05:12:39.689795 18931 webserver.cc:460] Webserver started at http://127.0.98.62:43463/ using document root <none> and password file <none>
I20260924 05:12:39.690038 18931 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:39.690104 18931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:39.690232 18931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:39.691258 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "23d264a37f474c7fbbb7527b0ea7a6d3"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "2df88b77ea1f3944b8324559202eb52c"
server_key_iv: "889401c5075f2b6d32742f5896628127"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.691588 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "23d264a37f474c7fbbb7527b0ea7a6d3"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "2df88b77ea1f3944b8324559202eb52c"
server_key_iv: "889401c5075f2b6d32742f5896628127"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.692821 18931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:12:39.693475 18945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.693635 18931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:39.693740 18931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "23d264a37f474c7fbbb7527b0ea7a6d3"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "2df88b77ea1f3944b8324559202eb52c"
server_key_iv: "889401c5075f2b6d32742f5896628127"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.693850 18931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:39.709651 18931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:39.709926 18931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:39.710072 18931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:39.713913 18931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42115
I20260924 05:12:39.713944 18997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42115 every 8 connection(s)
I20260924 05:12:39.714270 18931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:39.715240 18998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:39.716352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 18931
I20260924 05:12:39.716478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:39.716694 392 external_mini_cluster.cc:1428] Setting key 07d2a15dc035136e92186f730a049f06
I20260924 05:12:39.717975 18998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3: Bootstrap starting.
I20260924 05:12:39.718590 18998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:39.718914 18998 log.cc:824] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:39.719651 18998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3: No bootstrap required, opened a new log
I20260924 05:12:39.721807 18998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d264a37f474c7fbbb7527b0ea7a6d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260924 05:12:39.722007 18998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:39.722052 18998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d264a37f474c7fbbb7527b0ea7a6d3, State: Initialized, Role: FOLLOWER
I20260924 05:12:39.722172 18998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d264a37f474c7fbbb7527b0ea7a6d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260924 05:12:39.722241 18998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:39.722250 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.717204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.0.98.62:42115)
0924 05:12:39.717386 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.717400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.717423 (+ 23us) server_negotiation.cc:407] Connection header received
0924 05:12:39.717795 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.717801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.717857 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.717964 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:39.718206 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.719371 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.720443 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.720812 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.720899 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.721600 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:39.721604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:39.721604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:39.721752 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:39.721785 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:39.721795 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:39.721847 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.721949 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.722028 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260924 05:12:39.722303 18998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:39.722343 18998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:39.722985 18998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d264a37f474c7fbbb7527b0ea7a6d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260924 05:12:39.723091 18998 leader_election.cc:302] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23d264a37f474c7fbbb7527b0ea7a6d3; no voters:
I20260924 05:12:39.723242 18998 leader_election.cc:288] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:39.723265 19003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:39.723438 19003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [term 1 LEADER]: Becoming Leader. State: Replica: 23d264a37f474c7fbbb7527b0ea7a6d3, State: Running, Role: LEADER
I20260924 05:12:39.723469 18998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:39.723577 19003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d264a37f474c7fbbb7527b0ea7a6d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260924 05:12:39.723928 19004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23d264a37f474c7fbbb7527b0ea7a6d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d264a37f474c7fbbb7527b0ea7a6d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260924 05:12:39.723937 19005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23d264a37f474c7fbbb7527b0ea7a6d3. Latest consensus state: current_term: 1 leader_uuid: "23d264a37f474c7fbbb7527b0ea7a6d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d264a37f474c7fbbb7527b0ea7a6d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260924 05:12:39.724100 19004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:39.724107 19005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:39.724555 19012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:39.725008 19012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:39.726583 19012 catalog_manager.cc:1347] Generated new cluster ID: 9e357965a8524855b4fb0de66ac6c94f
I20260924 05:12:39.726625 19012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:39.737412 19012 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:39.737910 19012 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:39.741801 19012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3: Generated new TSK 0
I20260924 05:12:39.741989 19012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:39.743716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--builtin_ntp_servers=127.0.98.20:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:12:39.822255 19022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:39.822476 19022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:39.822512 19022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:39.823917 19022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:39.823982 19022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:39.824024 19022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:12:39.825536 19022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:39.825791 19022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:39.826084 19022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:39.830935 19030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.830979 19027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:39.831094 19022 server_base.cc:1034] running on GCE node
W20260924 05:12:39.830948 19028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:39.831414 19022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:39.831662 19022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:39.832841 19022 hybrid_clock.cc:648] HybridClock initialized: now 1790226759832827 us; error 28 us; skew 500 ppm
I20260924 05:12:39.834146 19022 webserver.cc:460] Webserver started at http://127.0.98.1:37101/ using document root <none> and password file <none>
I20260924 05:12:39.834405 19022 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:39.834465 19022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:39.834587 19022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:39.836027 19022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "2bc8178583f14862a7cd6d04a35cc41a"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "5662ab91eb91e5d6e4f57c9f51aec287"
server_key_iv: "566330248b7f5d6e69001978d64837f2"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.836444 19022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2bc8178583f14862a7cd6d04a35cc41a"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "5662ab91eb91e5d6e4f57c9f51aec287"
server_key_iv: "566330248b7f5d6e69001978d64837f2"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.838032 19022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:12:39.838817 19036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.839003 19022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:12:39.839098 19022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "2bc8178583f14862a7cd6d04a35cc41a"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "5662ab91eb91e5d6e4f57c9f51aec287"
server_key_iv: "566330248b7f5d6e69001978d64837f2"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.839185 19022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:39.871076 19022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:39.871407 19022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:39.871562 19022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:39.871803 19022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:39.872148 19022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:39.872211 19022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.872255 19022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:39.872282 19022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.878660 19022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35865
I20260924 05:12:39.878728 19149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35865 every 8 connection(s)
I20260924 05:12:39.879101 19022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:12:39.883858 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.879844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49129 (local address 127.0.98.62:42115)
0924 05:12:39.879908 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:39.879911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:39.880167 (+ 256us) server_negotiation.cc:407] Connection header received
0924 05:12:39.880625 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:39.880626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:39.880636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:39.880690 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:39.881299 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.881782 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.882683 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.882881 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.882932 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.883658 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:39.883659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:39.883660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:39.883664 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:39.883676 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:39.883679 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:39.883720 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:39.883780 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 05:12:39.883804 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:12:39.884089 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:39.879837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:49129)
0924 05:12:39.880046 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:39.880080 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:39.880542 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:39.880741 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:39.880745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:39.880752 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:39.881232 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.881235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.881868 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.881869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.882613 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:39.882615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:39.882952 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:39.882953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:39.883069 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:39.883533 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:12:39.883533 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:39.883559 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:12:39.883581 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:39.883720 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:39.883722 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:39.883724 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:12:39.883768 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:12:39.883847 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":119,"threads_started":1}
I20260924 05:12:39.884644 19150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260924 05:12:39.884773 19150 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:39.884951 19150 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260924 05:12:39.885444 18962 ts_manager.cc:194] Registered new tserver with Master: 2bc8178583f14862a7cd6d04a35cc41a (127.0.98.1:35865)
I20260924 05:12:39.886034 18962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49129
I20260924 05:12:39.888573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 19022
I20260924 05:12:39.888680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:12:39.888883 392 external_mini_cluster.cc:1428] Setting key 7c4881bbc1bbcffccedf56b57b84e8ad
I20260924 05:12:39.889714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--builtin_ntp_servers=127.0.98.20:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:12:39.968029 19153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:39.968218 19153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:39.968238 19153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:39.969319 19153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:39.969358 19153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:39.969408 19153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:12:39.970840 19153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:39.971068 19153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:39.971354 19153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:39.976209 19159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.976221 19161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:39.976224 19158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:39.976337 19153 server_base.cc:1034] running on GCE node
I20260924 05:12:39.976543 19153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:39.976789 19153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:39.977960 19153 hybrid_clock.cc:648] HybridClock initialized: now 1790226759977952 us; error 25 us; skew 500 ppm
I20260924 05:12:39.979167 19153 webserver.cc:460] Webserver started at http://127.0.98.2:36295/ using document root <none> and password file <none>
I20260924 05:12:39.979406 19153 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:39.979475 19153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:39.979598 19153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:39.980623 19153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "8f7a71316ce04b288b7e66841eed1682"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "62753232c3669c008c27272068a1bcd8"
server_key_iv: "db99be2b29c2fda268668265a2b91ce7"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.980978 19153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8f7a71316ce04b288b7e66841eed1682"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "62753232c3669c008c27272068a1bcd8"
server_key_iv: "db99be2b29c2fda268668265a2b91ce7"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.982214 19153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:12:39.982924 19167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:39.983088 19153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:39.983191 19153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "8f7a71316ce04b288b7e66841eed1682"
format_stamp: "Formatted at 2026-09-24 05:12:39 on dist-test-slave-mr2f"
server_key: "62753232c3669c008c27272068a1bcd8"
server_key_iv: "db99be2b29c2fda268668265a2b91ce7"
server_key_version: "encryptionkey@0"
I20260924 05:12:39.983280 19153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:40.002096 19153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:40.002372 19153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:40.002518 19153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:40.002717 19153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:40.003103 19153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:40.003157 19153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:40.003196 19153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:40.003221 19153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:40.009213 19153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34417
I20260924 05:12:40.009254 19280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34417 every 8 connection(s)
I20260924 05:12:40.009580 19153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:12:40.014182 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.010267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51323 (local address 127.0.98.62:42115)
0924 05:12:40.010313 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:40.010316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:40.010546 (+ 230us) server_negotiation.cc:407] Connection header received
0924 05:12:40.011011 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:40.011012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:40.011022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:40.011063 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:40.011647 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.012136 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.013043 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.013227 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.013275 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.013992 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:40.013993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:40.013993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:40.013997 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:40.014009 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:40.014013 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:40.014054 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:40.014109 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 05:12:40.014128 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:12:40.014410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 19153
I20260924 05:12:40.014449 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.010261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:51323)
0924 05:12:40.010437 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:40.010467 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:40.010929 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:40.011113 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:40.011116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:40.011123 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:40.011581 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.011585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.012213 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.012213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.012974 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.012976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.013295 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.013296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.013394 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.013864 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:12:40.013864 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:40.013903 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:12:40.013922 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:40.014062 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:40.014065 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:40.014067 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:12:40.014110 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:12:40.014199 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260924 05:12:40.014513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:12:40.014748 392 external_mini_cluster.cc:1428] Setting key 485f1818e94cb62aa60d0d0a428b96f2
I20260924 05:12:40.014977 19281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260924 05:12:40.015085 19281 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:40.015257 19281 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260924 05:12:40.015569 18962 ts_manager.cc:194] Registered new tserver with Master: 8f7a71316ce04b288b7e66841eed1682 (127.0.98.2:34417)
I20260924 05:12:40.015770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--builtin_ntp_servers=127.0.98.20:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:12:40.016018 18962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51323
W20260924 05:12:40.092348 19284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:40.092525 19284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:40.092545 19284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:40.093649 19284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:40.093686 19284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:40.093736 19284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:12:40.095013 19284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:40.095237 19284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:40.095501 19284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:40.100356 19290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:40.100378 19289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:40.100383 19292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:40.100358 19284 server_base.cc:1034] running on GCE node
I20260924 05:12:40.100752 19284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:40.100987 19284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:40.102137 19284 hybrid_clock.cc:648] HybridClock initialized: now 1790226760102107 us; error 28 us; skew 500 ppm
I20260924 05:12:40.103338 19284 webserver.cc:460] Webserver started at http://127.0.98.3:34897/ using document root <none> and password file <none>
I20260924 05:12:40.103597 19284 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:40.103665 19284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:40.103817 19284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:40.104809 19284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "0ad5848dece74bcca1b1e974401f6821"
format_stamp: "Formatted at 2026-09-24 05:12:40 on dist-test-slave-mr2f"
server_key: "63872acd377f0f62637577c40ba9346e"
server_key_iv: "8369ec861ea9e96ff9b5994f5364829b"
server_key_version: "encryptionkey@0"
I20260924 05:12:40.105151 19284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0ad5848dece74bcca1b1e974401f6821"
format_stamp: "Formatted at 2026-09-24 05:12:40 on dist-test-slave-mr2f"
server_key: "63872acd377f0f62637577c40ba9346e"
server_key_iv: "8369ec861ea9e96ff9b5994f5364829b"
server_key_version: "encryptionkey@0"
I20260924 05:12:40.106400 19284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:12:40.107141 19298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:40.107302 19284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:40.107411 19284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "0ad5848dece74bcca1b1e974401f6821"
format_stamp: "Formatted at 2026-09-24 05:12:40 on dist-test-slave-mr2f"
server_key: "63872acd377f0f62637577c40ba9346e"
server_key_iv: "8369ec861ea9e96ff9b5994f5364829b"
server_key_version: "encryptionkey@0"
I20260924 05:12:40.107492 19284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:40.120849 19284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:40.121130 19284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:40.121268 19284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:40.121477 19284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:40.121773 19284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:40.121824 19284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:40.121866 19284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:40.121892 19284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:40.127803 19284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45133
I20260924 05:12:40.127843 19411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45133 every 8 connection(s)
I20260924 05:12:40.128149 19284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:12:40.129786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 19284
I20260924 05:12:40.129869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:12:40.130075 392 external_mini_cluster.cc:1428] Setting key 49ad00e71d552548495f5dee21831e44
I20260924 05:12:40.133250 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.128825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59381 (local address 127.0.98.62:42115)
0924 05:12:40.128892 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:40.128894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:40.129113 (+ 219us) server_negotiation.cc:407] Connection header received
0924 05:12:40.129513 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:40.129514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:40.129524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:40.129573 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:40.130397 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.131149 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.132055 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.132232 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.132282 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.133055 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:40.133056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:40.133056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:40.133060 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:40.133082 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:40.133085 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:40.133120 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:40.133178 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 05:12:40.133196 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:12:40.133466 19414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.128814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:59381)
0924 05:12:40.128992 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:40.129022 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:40.129439 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:40.129696 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:40.129698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:40.129706 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:40.130310 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.130316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.131230 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.131231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.131986 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.131988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.132303 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.132303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.132410 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.132929 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:12:40.132930 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:40.132963 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:12:40.132984 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:40.133122 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:40.133124 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:40.133126 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:12:40.133169 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:12:40.133236 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 05:12:40.133860 19412 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260924 05:12:40.133941 19412 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:40.134071 19412 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260924 05:12:40.134377 18962 ts_manager.cc:194] Registered new tserver with Master: 0ad5848dece74bcca1b1e974401f6821 (127.0.98.3:45133)
I20260924 05:12:40.134716 18962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59381
I20260924 05:12:40.141790 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:12:40.146034 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.143144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.0.98.62:42115)
0924 05:12:40.143225 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:40.143227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:40.143315 (+ 88us) server_negotiation.cc:407] Connection header received
0924 05:12:40.143416 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:40.143417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:40.143426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:40.143480 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:40.143639 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.144097 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.144875 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.145078 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.145148 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.145729 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:40.145730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:40.145731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:40.145735 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:40.145747 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:40.145751 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:40.145863 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:40.145950 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:12:40.145959 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 05:12:40.147480 18962 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:12:40.148926 18962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:12:40.154863 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.151251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.0.98.3:45133)
0924 05:12:40.151402 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:40.151405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:40.151413 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:12:40.151443 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:40.151446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:40.151458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:40.151522 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:40.151689 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.152443 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.154029 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.154458 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.154541 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.154620 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:40.154719 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 05:12:40.154781 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260924 05:12:40.154903 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.150904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35865 (local address 127.0.0.1:41660)
0924 05:12:40.151087 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:40.151124 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:40.151191 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:40.151521 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:40.151524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:40.151526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:40.151593 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.151614 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.152949 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.152950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.153743 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.153745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.154600 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.154601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.154660 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.154665 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:12:40.154697 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 05:12:40.154790 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260924 05:12:40.154902 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.151101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45133 (local address 127.0.0.1:59690)
0924 05:12:40.151212 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:40.151222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:40.151271 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:40.151573 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:40.151573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:40.151575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:40.151616 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.151618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.152530 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.152531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.153679 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.153682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.154553 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.154554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.154625 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.154637 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 05:12:40.154705 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:12:40.154793 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":24,"thread_start_us":52,"threads_started":1}
I20260924 05:12:40.155164 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.151001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.0.98.1:35865)
0924 05:12:40.151339 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:40.151343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:40.151354 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:12:40.151403 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:40.151405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:40.151422 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:40.151507 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:40.151698 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.152839 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.153835 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.154503 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.154942 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.155025 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:40.155047 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 05:12:40.155080 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":80,"threads_started":1}
I20260924 05:12:40.155097 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.150946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34417 (local address 127.0.0.1:37292)
0924 05:12:40.151162 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:40.151170 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:40.151210 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:40.151600 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:40.151601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:40.151603 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:40.151650 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.151651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.152931 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.152932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.153957 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.153959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.154897 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.154898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.154954 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.154957 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:40.155003 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 05:12:40.155039 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":13,"thread_start_us":77,"threads_started":1}
I20260924 05:12:40.155299 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.151051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.0.98.2:34417)
0924 05:12:40.151387 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:40.151391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:40.151401 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:12:40.151439 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:40.151441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:40.151452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:40.151539 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:40.151722 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.152841 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.154033 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.154701 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.154780 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.154864 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:40.155159 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 05:12:40.155206 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":69,"threads_started":1}
I20260924 05:12:40.155903 19346 tablet_service.cc:1467] Processing CreateTablet for tablet 9edabb06965645fabf7bae3775a17b98 (DEFAULT_TABLE table=test-table [id=4d8f4917fb5242eba4177c2b02a8662a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:40.155995 19215 tablet_service.cc:1467] Processing CreateTablet for tablet 9edabb06965645fabf7bae3775a17b98 (DEFAULT_TABLE table=test-table [id=4d8f4917fb5242eba4177c2b02a8662a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:40.156251 19215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edabb06965645fabf7bae3775a17b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:40.156251 19346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edabb06965645fabf7bae3775a17b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:40.156216 19084 tablet_service.cc:1467] Processing CreateTablet for tablet 9edabb06965645fabf7bae3775a17b98 (DEFAULT_TABLE table=test-table [id=4d8f4917fb5242eba4177c2b02a8662a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:40.156476 19084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edabb06965645fabf7bae3775a17b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:40.158877 19426 tablet_bootstrap.cc:492] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a: Bootstrap starting.
I20260924 05:12:40.159385 19427 tablet_bootstrap.cc:492] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: Bootstrap starting.
I20260924 05:12:40.159536 19426 tablet_bootstrap.cc:654] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:40.159859 19426 log.cc:824] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:40.160028 19427 tablet_bootstrap.cc:654] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:40.160204 19428 tablet_bootstrap.cc:492] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: Bootstrap starting.
I20260924 05:12:40.160337 19427 log.cc:824] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:40.160620 19426 tablet_bootstrap.cc:492] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a: No bootstrap required, opened a new log
I20260924 05:12:40.160737 19426 ts_tablet_manager.cc:1397] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:12:40.160784 19428 tablet_bootstrap.cc:654] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:40.161067 19427 tablet_bootstrap.cc:492] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: No bootstrap required, opened a new log
I20260924 05:12:40.161115 19428 log.cc:824] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:40.161166 19427 ts_tablet_manager.cc:1397] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:12:40.161967 19428 tablet_bootstrap.cc:492] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: No bootstrap required, opened a new log
I20260924 05:12:40.162047 19428 ts_tablet_manager.cc:1397] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:12:40.163007 19426 raft_consensus.cc:348] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } }
I20260924 05:12:40.163152 19426 raft_consensus.cc:374] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:40.163213 19426 raft_consensus.cc:729] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc8178583f14862a7cd6d04a35cc41a, State: Initialized, Role: FOLLOWER
I20260924 05:12:40.163336 19426 consensus_queue.cc:260] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } }
I20260924 05:12:40.163611 19426 ts_tablet_manager.cc:1428] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:12:40.163719 19150 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260924 05:12:40.163908 19427 raft_consensus.cc:348] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } }
I20260924 05:12:40.164074 19427 raft_consensus.cc:374] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:40.164109 19427 raft_consensus.cc:729] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad5848dece74bcca1b1e974401f6821, State: Initialized, Role: FOLLOWER
I20260924 05:12:40.164207 19427 consensus_queue.cc:260] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } }
I20260924 05:12:40.164435 19427 ts_tablet_manager.cc:1428] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:12:40.164549 19412 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260924 05:12:40.164640 19428 raft_consensus.cc:348] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } }
I20260924 05:12:40.164808 19428 raft_consensus.cc:374] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:40.164860 19428 raft_consensus.cc:729] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f7a71316ce04b288b7e66841eed1682, State: Initialized, Role: FOLLOWER
I20260924 05:12:40.164971 19428 consensus_queue.cc:260] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } }
I20260924 05:12:40.165220 19428 ts_tablet_manager.cc:1428] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:12:40.165258 19281 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260924 05:12:40.211642 19432 raft_consensus.cc:482] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:12:40.211760 19432 raft_consensus.cc:504] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } }
I20260924 05:12:40.212168 19432 leader_election.cc:288] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ad5848dece74bcca1b1e974401f6821 (127.0.98.3:45133), 8f7a71316ce04b288b7e66841eed1682 (127.0.98.2:34417)
I20260924 05:12:40.214955 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.212462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47573 (local address 127.0.98.3:45133)
0924 05:12:40.212508 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:40.212510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:40.212519 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:12:40.212555 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:40.212556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:40.212563 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:40.212638 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:40.212797 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.213278 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.214199 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.214547 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.214640 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.214650 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:40.214839 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 05:12:40.214874 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:12:40.214972 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.212463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54413 (local address 127.0.98.2:34417)
0924 05:12:40.212508 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:40.212510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:40.212626 (+ 116us) server_negotiation.cc:407] Connection header received
0924 05:12:40.212687 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:40.212688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:40.212698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:40.212773 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:40.212931 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.213380 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.214279 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.214632 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.214715 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.214724 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:40.214867 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 05:12:40.214903 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:12:40.214987 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.212327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45133 (local address 127.0.98.1:47573)
0924 05:12:40.212381 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:40.212393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:40.212443 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:40.212643 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:40.212645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:40.212646 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:40.212717 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.212718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.213363 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.213364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.214104 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.214106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.214641 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.214642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.214712 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.214715 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:40.214831 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 05:12:40.214874 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":2}
I20260924 05:12:40.215024 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.212361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34417 (local address 127.0.98.1:54413)
0924 05:12:40.212555 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:40.212563 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:40.212630 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:40.212776 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:40.212776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:40.212778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:40.212852 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.212853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.213460 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.213462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.214187 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:40.214189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.214716 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.214717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:40.214810 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.214813 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:40.214865 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:12:40.214900 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":115,"threads_started":1}
I20260924 05:12:40.215348 19366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edabb06965645fabf7bae3775a17b98" candidate_uuid: "2bc8178583f14862a7cd6d04a35cc41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad5848dece74bcca1b1e974401f6821" is_pre_election: true
I20260924 05:12:40.215490 19366 raft_consensus.cc:2454] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bc8178583f14862a7cd6d04a35cc41a in term 0.
I20260924 05:12:40.215425 19235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edabb06965645fabf7bae3775a17b98" candidate_uuid: "2bc8178583f14862a7cd6d04a35cc41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7a71316ce04b288b7e66841eed1682" is_pre_election: true
I20260924 05:12:40.215587 19235 raft_consensus.cc:2454] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bc8178583f14862a7cd6d04a35cc41a in term 0.
I20260924 05:12:40.215752 19037 leader_election.cc:302] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad5848dece74bcca1b1e974401f6821, 2bc8178583f14862a7cd6d04a35cc41a; no voters:
I20260924 05:12:40.215902 19432 raft_consensus.cc:2781] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:12:40.215958 19432 raft_consensus.cc:482] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:12:40.215987 19432 raft_consensus.cc:3037] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:40.216605 19432 raft_consensus.cc:504] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } }
I20260924 05:12:40.216737 19432 leader_election.cc:288] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [CANDIDATE]: Term 1 election: Requested vote from peers 0ad5848dece74bcca1b1e974401f6821 (127.0.98.3:45133), 8f7a71316ce04b288b7e66841eed1682 (127.0.98.2:34417)
I20260924 05:12:40.216856 19366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edabb06965645fabf7bae3775a17b98" candidate_uuid: "2bc8178583f14862a7cd6d04a35cc41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad5848dece74bcca1b1e974401f6821"
I20260924 05:12:40.216874 19235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edabb06965645fabf7bae3775a17b98" candidate_uuid: "2bc8178583f14862a7cd6d04a35cc41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7a71316ce04b288b7e66841eed1682"
I20260924 05:12:40.216952 19366 raft_consensus.cc:3037] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:40.216950 19235 raft_consensus.cc:3037] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:40.217669 19366 raft_consensus.cc:2454] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bc8178583f14862a7cd6d04a35cc41a in term 1.
I20260924 05:12:40.217716 19235 raft_consensus.cc:2454] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bc8178583f14862a7cd6d04a35cc41a in term 1.
I20260924 05:12:40.217859 19037 leader_election.cc:302] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad5848dece74bcca1b1e974401f6821, 2bc8178583f14862a7cd6d04a35cc41a; no voters:
I20260924 05:12:40.217978 19432 raft_consensus.cc:2781] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:40.218163 19432 raft_consensus.cc:686] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 1 LEADER]: Becoming Leader. State: Replica: 2bc8178583f14862a7cd6d04a35cc41a, State: Running, Role: LEADER
I20260924 05:12:40.218273 19432 consensus_queue.cc:237] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } }
I20260924 05:12:40.219170 18961 catalog_manager.cc:5495] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bc8178583f14862a7cd6d04a35cc41a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2bc8178583f14862a7cd6d04a35cc41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bc8178583f14862a7cd6d04a35cc41a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 } health_report { overall_health: UNKNOWN } } }
I20260924 05:12:40.227762 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.225353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.0.98.1:35865)
0924 05:12:40.225419 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:40.225421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:40.225430 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:12:40.225505 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:40.225505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:40.225523 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:40.225592 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:12:40.225716 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.226156 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.226972 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:40.227169 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:40.227250 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:40.227411 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:12:40.227512 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:12:40.227625 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:40.227655 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 05:12:40.227665 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 05:12:40.229339 19235 raft_consensus.cc:1261] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [term 1 FOLLOWER]: Refusing update from remote peer 2bc8178583f14862a7cd6d04a35cc41a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:40.229413 19366 raft_consensus.cc:1261] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [term 1 FOLLOWER]: Refusing update from remote peer 2bc8178583f14862a7cd6d04a35cc41a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:40.229744 19436 consensus_queue.cc:1035] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad5848dece74bcca1b1e974401f6821" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:40.229876 19432 consensus_queue.cc:1035] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f7a71316ce04b288b7e66841eed1682" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:40.232906 19441 mvcc.cc:205] Tried to move back new op lower bound from 7332768809897029632 to 7332768809854779392. Current Snapshot: MvccSnapshot[applied={T|T < 7332768809897029632 or (T in {7332768809897029632})}]
I20260924 05:12:40.237673 18961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:40.237756 18961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:40.239869 18961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23d264a37f474c7fbbb7527b0ea7a6d3: Sending DeleteTablet for 3 replicas of tablet 9edabb06965645fabf7bae3775a17b98
I20260924 05:12:40.240373 19084 tablet_service.cc:1514] Processing DeleteTablet for tablet 9edabb06965645fabf7bae3775a17b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:40 UTC) from {username='slave'} at 127.0.0.1:41660
I20260924 05:12:40.240371 19346 tablet_service.cc:1514] Processing DeleteTablet for tablet 9edabb06965645fabf7bae3775a17b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:40 UTC) from {username='slave'} at 127.0.0.1:59690
I20260924 05:12:40.240404 19215 tablet_service.cc:1514] Processing DeleteTablet for tablet 9edabb06965645fabf7bae3775a17b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:40 UTC) from {username='slave'} at 127.0.0.1:37292
I20260924 05:12:40.240600 19448 tablet_replica.cc:330] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a: stopping tablet replica
I20260924 05:12:40.240701 19448 raft_consensus.cc:2229] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:12:40.240780 19449 tablet_replica.cc:330] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: stopping tablet replica
I20260924 05:12:40.240846 19448 raft_consensus.cc:2258] T 9edabb06965645fabf7bae3775a17b98 P 2bc8178583f14862a7cd6d04a35cc41a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:40.240929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 19022
I20260924 05:12:40.240933 19449 raft_consensus.cc:2229] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:12:40.242501 19449 raft_consensus.cc:2258] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:40.242749 19450 tablet_replica.cc:330] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: stopping tablet replica
I20260924 05:12:40.242870 19450 raft_consensus.cc:2229] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:12:40.242995 19450 raft_consensus.cc:2258] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:40.243304 19449 ts_tablet_manager.cc:1905] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:40.244688 19450 ts_tablet_manager.cc:1905] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:40.244810 19449 ts_tablet_manager.cc:1918] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:40.244908 19449 log.cc:1196] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/9edabb06965645fabf7bae3775a17b98
I20260924 05:12:40.245169 19449 ts_tablet_manager.cc:1939] T 9edabb06965645fabf7bae3775a17b98 P 0ad5848dece74bcca1b1e974401f6821: Deleting consensus metadata
I20260924 05:12:40.245576 18946 catalog_manager.cc:4841] TS 0ad5848dece74bcca1b1e974401f6821 (127.0.98.3:45133): tablet 9edabb06965645fabf7bae3775a17b98 (table test-table [id=4d8f4917fb5242eba4177c2b02a8662a]) successfully deleted
W20260924 05:12:40.246245 18946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35865 (error 108)
I20260924 05:12:40.246428 19450 ts_tablet_manager.cc:1918] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:40.246495 19450 log.cc:1196] T 9edabb06965645fabf7bae3775a17b98 P 8f7a71316ce04b288b7e66841eed1682: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/9edabb06965645fabf7bae3775a17b98
I20260924 05:12:40.246490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 19153
I20260924 05:12:40.248965 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.248819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35865 (local address 127.0.0.1:41678)
0924 05:12:40.248882 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:40.248925 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 05:12:40.249105 18946 catalog_manager.cc:4584] TS 2bc8178583f14862a7cd6d04a35cc41a (127.0.98.1:35865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9edabb06965645fabf7bae3775a17b98: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35865: connect: Connection refused (error 111)
I20260924 05:12:40.251183 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:40.251076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34417 (local address 127.0.0.1:37296)
0924 05:12:40.251130 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:40.251156 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 05:12:40.251192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 19284
I20260924 05:12:40.255867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 18931
2026-09-24T05:12:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (680 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:40 dist-test-slave-mr2f krb5kdc[19454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:40 dist-test-slave-mr2f krb5kdc[19454](info): set up 2 sockets
Sep 24 05:12:40 dist-test-slave-mr2f krb5kdc[19454](info): commencing operation
krb5kdc: starting...
W20260924 05:12:41.274573 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:41 dist-test-slave-mr2f krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:12:41.385170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:41.463737 19467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:41.463989 19467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:41.464026 19467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:41.465188 19467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:41.465256 19467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:41.465283 19467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:41.465315 19467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:41.465339 19467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:41.466893 19467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:41.467187 19467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:41.467410 19467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:12:41.472040 19467 server_base.cc:1034] running on GCE node
W20260924 05:12:41.472174 19475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:41.472203 19473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:41.472222 19472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:41.472594 19467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:41.472867 19467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:41.474048 19467 hybrid_clock.cc:648] HybridClock initialized: now 1790226761474042 us; error 37 us; skew 500 ppm
Sep 24 05:12:41 dist-test-slave-mr2f krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:41.476378 19467 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:12:41.476819 19467 webserver.cc:460] Webserver started at http://127.0.98.62:43071/ using document root <none> and password file <none>
I20260924 05:12:41.477059 19467 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:41.477123 19467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:41.477267 19467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:41.478240 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "45355b28f37e47d0a4952fd370371ce8"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "444c4718658d384cdec8a22a7e1456e9"
server_key_iv: "69839a245a5a2c4cdef14d0c87ba6fd7"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.478593 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "45355b28f37e47d0a4952fd370371ce8"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "444c4718658d384cdec8a22a7e1456e9"
server_key_iv: "69839a245a5a2c4cdef14d0c87ba6fd7"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.479903 19467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:12:41.480583 19482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.480749 19467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:12:41.480842 19467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "45355b28f37e47d0a4952fd370371ce8"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "444c4718658d384cdec8a22a7e1456e9"
server_key_iv: "69839a245a5a2c4cdef14d0c87ba6fd7"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.480933 19467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:41.496147 19467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:41.497844 19467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:41.498005 19467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:41.501906 19467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38289
I20260924 05:12:41.501933 19534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38289 every 8 connection(s)
I20260924 05:12:41.502283 19467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:41.503096 19535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:41.505167 19535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8: Bootstrap starting.
I20260924 05:12:41.505748 19535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:41.505976 19535 log.cc:824] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:41.506589 19535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8: No bootstrap required, opened a new log
I20260924 05:12:41.508149 19535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45355b28f37e47d0a4952fd370371ce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260924 05:12:41.508281 19535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:41.508303 19535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45355b28f37e47d0a4952fd370371ce8, State: Initialized, Role: FOLLOWER
I20260924 05:12:41.508438 19535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45355b28f37e47d0a4952fd370371ce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260924 05:12:41.508520 19535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:41.508543 19535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:41.508570 19535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:41.509042 19535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45355b28f37e47d0a4952fd370371ce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260924 05:12:41.509119 19535 leader_election.cc:302] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45355b28f37e47d0a4952fd370371ce8; no voters:
I20260924 05:12:41.509228 19535 leader_election.cc:288] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:41.509299 19538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:41.509447 19538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [term 1 LEADER]: Becoming Leader. State: Replica: 45355b28f37e47d0a4952fd370371ce8, State: Running, Role: LEADER
I20260924 05:12:41.509498 19535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:41.509531 19538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45355b28f37e47d0a4952fd370371ce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260924 05:12:41.509893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 19467
I20260924 05:12:41.509876 19540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45355b28f37e47d0a4952fd370371ce8. Latest consensus state: current_term: 1 leader_uuid: "45355b28f37e47d0a4952fd370371ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45355b28f37e47d0a4952fd370371ce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } } }
I20260924 05:12:41.509954 19540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:41.510020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:41.509999 19539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45355b28f37e47d0a4952fd370371ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45355b28f37e47d0a4952fd370371ce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } } }
I20260924 05:12:41.510054 19539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:41.510260 392 external_mini_cluster.cc:1428] Setting key 6e666d324fa71266f4e28800543e7cc3
I20260924 05:12:41.510702 19544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:41.511317 19544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:41.512743 19544 catalog_manager.cc:1347] Generated new cluster ID: 90b8e4be6cad4191a2f3fc7336a95bf8
I20260924 05:12:41.512792 19544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:12:41 dist-test-slave-mr2f krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:41.520222 19544 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:41.520871 19544 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:41.521682 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.510902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.0.98.62:38289)
0924 05:12:41.511112 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:41.511121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:41.511155 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:12:41.511318 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:41.511325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:41.511393 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:41.511535 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:41.512295 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.513589 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.514648 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.515234 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.515344 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.518000 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:41.518006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:41.518006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:41.518021 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:41.520022 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.520465 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.520469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.520469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.520588 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.520805 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.520805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.520806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.521068 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:41.521215 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:41.521325 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 05:12:41.521479 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260924 05:12:41.530942 19544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8: Generated new TSK 0
I20260924 05:12:41.531113 19544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:12:41.561290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:41.639688 19561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:41.639895 19561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:41.639920 19561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:41.641088 19561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:41.641140 19561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:41.641166 19561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:12:41.642724 19561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:41.642976 19561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:41.643227 19561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:41.648080 19566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:41.648133 19567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:41.648273 19561 server_base.cc:1034] running on GCE node
W20260924 05:12:41.648077 19569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:41.648543 19561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:41.648782 19561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:41.649942 19561 hybrid_clock.cc:648] HybridClock initialized: now 1790226761649920 us; error 35 us; skew 500 ppm
Sep 24 05:12:41 dist-test-slave-mr2f krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:41.652347 19561 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:12:41.652750 19561 webserver.cc:460] Webserver started at http://127.0.98.1:33737/ using document root <none> and password file <none>
I20260924 05:12:41.652988 19561 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:41.653054 19561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:41.653203 19561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:41.654218 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d94d09421d04aa3b57eb0a2310ff4c4"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "bce8067470c6e08529995716071277f5"
server_key_iv: "2e3116f4dc52c543d0e6db2c69f56ce7"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.654558 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d94d09421d04aa3b57eb0a2310ff4c4"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "bce8067470c6e08529995716071277f5"
server_key_iv: "2e3116f4dc52c543d0e6db2c69f56ce7"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.655977 19561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:12:41.656622 19576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.656770 19561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.656872 19561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "6d94d09421d04aa3b57eb0a2310ff4c4"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "bce8067470c6e08529995716071277f5"
server_key_iv: "2e3116f4dc52c543d0e6db2c69f56ce7"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.656953 19561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:41.671744 19561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:41.673429 19561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:41.673583 19561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:41.673789 19561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:41.674084 19561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:41.674137 19561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.674176 19561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:41.674206 19561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.679814 19561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40303
I20260924 05:12:41.679849 19689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40303 every 8 connection(s)
I20260924 05:12:41.680176 19561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:12:41.685839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 19561
Sep 24 05:12:41 dist-test-slave-mr2f krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:41.685967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:12:41.686162 392 external_mini_cluster.cc:1428] Setting key 96c22c5e5aeccaaf03b37d3c2d385ddf
I20260924 05:12:41.689231 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.680858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51983 (local address 127.0.98.62:38289)
0924 05:12:41.680930 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:41.680932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:41.681111 (+ 179us) server_negotiation.cc:407] Connection header received
0924 05:12:41.681984 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:41.681985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:41.681994 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:41.682042 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:41.682587 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.683155 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.684179 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.684376 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.684429 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.686408 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:41.686409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:41.686410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:41.686417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:41.687932 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.688298 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.688299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.688300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.688372 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.688589 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.688591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.688591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.688735 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:41.688807 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:41.689105 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 05:12:41.689172 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:12:41.689481 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.680846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38289 (local address 127.0.98.1:51983)
0924 05:12:41.680992 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:41.681025 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:41.681896 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:41.682108 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:41.682111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:41.682341 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:41.682516 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.682520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.683229 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.683230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.684107 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.684110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.684446 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.684447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.684543 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.684949 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:12:41.684956 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:41.686314 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:41.688020 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:41.688025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:41.688027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:41.688224 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:41.688439 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:41.688440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:41.688440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:41.688527 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:41.688807 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:41.688811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:41.688974 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 05:12:41.689076 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 05:12:41.689234 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260924 05:12:41.689955 19690 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38289
I20260924 05:12:41.690047 19690 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:41.690233 19690 heartbeater.cc:511] Master 127.0.98.62:38289 requested a full tablet report, sending...
I20260924 05:12:41.690735 19499 ts_manager.cc:194] Registered new tserver with Master: 6d94d09421d04aa3b57eb0a2310ff4c4 (127.0.98.1:40303)
I20260924 05:12:41.691375 19499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51983
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:12:41.709627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:41.787494 19695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:41.787680 19695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:41.787703 19695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:41.788815 19695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:41.788853 19695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:41.788879 19695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:12:41.790315 19695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:41.790477 19695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:41.790688 19695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:12:41.795425 19695 server_base.cc:1034] running on GCE node
W20260924 05:12:41.795411 19701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:41.795393 19700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:41.795389 19703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:41.795814 19695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:41.796090 19695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:41.797230 19695 hybrid_clock.cc:648] HybridClock initialized: now 1790226761797215 us; error 26 us; skew 500 ppm
Sep 24 05:12:41 dist-test-slave-mr2f krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:41.799742 19695 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:12:41.800128 19695 webserver.cc:460] Webserver started at http://127.0.98.2:44983/ using document root <none> and password file <none>
I20260924 05:12:41.800344 19695 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:41.800392 19695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:41.800552 19695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:41.801513 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "bbf0aae1a4294548a6c0bcaca67e3c79"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "20e67b638f199ae1503c929461222f3b"
server_key_iv: "f2996cc7f6b64b69ac31f073a0093525"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.801854 19695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bbf0aae1a4294548a6c0bcaca67e3c79"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "20e67b638f199ae1503c929461222f3b"
server_key_iv: "f2996cc7f6b64b69ac31f073a0093525"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.803144 19695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 05:12:41.803853 19710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.804028 19695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:41.804088 19695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "bbf0aae1a4294548a6c0bcaca67e3c79"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "20e67b638f199ae1503c929461222f3b"
server_key_iv: "f2996cc7f6b64b69ac31f073a0093525"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.804174 19695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:41.810999 19695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:41.812835 19695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:41.812940 19695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:41.813084 19695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:41.813303 19695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:41.813333 19695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.813352 19695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:41.813364 19695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.820060 19695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35099
I20260924 05:12:41.820138 19823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35099 every 8 connection(s)
I20260924 05:12:41.820461 19695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:12:41.823808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 19695
I20260924 05:12:41.823937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:12:41.824120 392 external_mini_cluster.cc:1428] Setting key 0acc5149a533b0cb7a16b8be4b080511
Sep 24 05:12:41 dist-test-slave-mr2f krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:41.832070 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.821230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42401 (local address 127.0.98.62:38289)
0924 05:12:41.821300 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:41.821302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:41.821494 (+ 192us) server_negotiation.cc:407] Connection header received
0924 05:12:41.822863 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:41.822864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:41.822877 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:41.822942 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:41.823773 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.824531 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.826276 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.826603 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.826683 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.829136 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:41.829137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:41.829138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:41.829144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:41.830527 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.831069 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.831070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.831070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.831121 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.831390 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.831391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.831391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.831486 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:41.831531 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:41.831897 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 05:12:41.831989 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 05:12:41.832275 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.821188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38289 (local address 127.0.98.2:42401)
0924 05:12:41.821370 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:41.821416 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:41.822659 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:41.823061 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:41.823065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:41.823365 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:41.823649 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.823655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.824855 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.824856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.826177 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.826180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.826692 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.826693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.826856 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.827343 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:12:41.827356 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:41.829042 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:41.830613 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:41.830619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:41.830622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:41.830984 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:41.831188 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:41.831188 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:41.831189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:41.831320 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:41.831549 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:41.831555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:41.831719 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 05:12:41.831838 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 05:12:41.832041 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 05:12:41.832787 19824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38289
I20260924 05:12:41.832903 19824 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:41.833073 19824 heartbeater.cc:511] Master 127.0.98.62:38289 requested a full tablet report, sending...
I20260924 05:12:41.833453 19499 ts_manager.cc:194] Registered new tserver with Master: bbf0aae1a4294548a6c0bcaca67e3c79 (127.0.98.2:35099)
I20260924 05:12:41.833771 19499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:12:41.847517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:41.928452 19829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:41.928668 19829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:41.928735 19829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:41.929881 19829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:41.929946 19829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:41.929991 19829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:12:41.931480 19829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:41.931722 19829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:41.931928 19829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:41.936687 19835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:41.936687 19837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:41.936827 19829 server_base.cc:1034] running on GCE node
W20260924 05:12:41.936983 19834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:41.937223 19829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:41.937495 19829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:41.938632 19829 hybrid_clock.cc:648] HybridClock initialized: now 1790226761938623 us; error 26 us; skew 500 ppm
Sep 24 05:12:41 dist-test-slave-mr2f krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:41.940995 19829 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:12:41.941429 19829 webserver.cc:460] Webserver started at http://127.0.98.3:32901/ using document root <none> and password file <none>
I20260924 05:12:41.941596 19829 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:41.941635 19829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:41.941720 19829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:41.942631 19829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4d7b36ea86440aaa4539418aaad2e71"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "95837c1189ccf13fd1741ade68c12ba1"
server_key_iv: "e771cb08b4e0ae7c9a158e369f97de6c"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.943042 19829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4d7b36ea86440aaa4539418aaad2e71"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "95837c1189ccf13fd1741ade68c12ba1"
server_key_iv: "e771cb08b4e0ae7c9a158e369f97de6c"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.944514 19829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:12:41.945416 19844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.945569 19829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:12:41.945663 19829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "d4d7b36ea86440aaa4539418aaad2e71"
format_stamp: "Formatted at 2026-09-24 05:12:41 on dist-test-slave-mr2f"
server_key: "95837c1189ccf13fd1741ade68c12ba1"
server_key_iv: "e771cb08b4e0ae7c9a158e369f97de6c"
server_key_version: "encryptionkey@0"
I20260924 05:12:41.945735 19829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:41.953637 19829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:41.955530 19829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:41.955689 19829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:41.955901 19829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:41.956193 19829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:41.956244 19829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.956285 19829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:41.956312 19829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:41.962347 19829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45177
I20260924 05:12:41.962387 19957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45177 every 8 connection(s)
I20260924 05:12:41.962689 19829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:12:41 dist-test-slave-mr2f krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:41.971223 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.963451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52997 (local address 127.0.98.62:38289)
0924 05:12:41.963499 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:41.963500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:41.963691 (+ 191us) server_negotiation.cc:407] Connection header received
0924 05:12:41.964577 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:41.964578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:41.964587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:41.964651 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:41.965204 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.965682 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.966640 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.966891 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.966943 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.968960 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:41.968962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:41.968962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:41.968968 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:41.970075 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.970438 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.970439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.970439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.970484 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.970686 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.970686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.970687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.970829 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:41.970898 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:41.971107 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 05:12:41.971163 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:12:41.971449 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.963419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38289 (local address 127.0.98.3:52997)
0924 05:12:41.963565 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:41.963606 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:41.964493 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:41.964704 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:41.964707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:41.964952 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:41.965136 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.965140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.965756 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.965757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.966563 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.966564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.966958 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.966958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.967055 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.967585 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:12:41.967593 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:41.968880 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:41.970142 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:41.970147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:41.970149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:41.970347 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:41.970543 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:12:41.970544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:12:41.970544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:12:41.970633 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:12:41.970892 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:41.970895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:41.971001 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 05:12:41.971079 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 05:12:41.971192 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260924 05:12:41.971866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 19829
I20260924 05:12:41.972002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:12:41.971990 19958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38289
I20260924 05:12:41.972074 19958 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:41.972209 19958 heartbeater.cc:511] Master 127.0.98.62:38289 requested a full tablet report, sending...
I20260924 05:12:41.972216 392 external_mini_cluster.cc:1428] Setting key bfa9563ba3e6db15fb5e30f442eb018b
I20260924 05:12:41.972553 19499 ts_manager.cc:194] Registered new tserver with Master: d4d7b36ea86440aaa4539418aaad2e71 (127.0.98.3:45177)
I20260924 05:12:41.973026 19499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52997
I20260924 05:12:41.973079 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:12:41.980017 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.974336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.0.98.62:38289)
0924 05:12:41.974377 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:41.974379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:41.974518 (+ 139us) server_negotiation.cc:407] Connection header received
0924 05:12:41.974581 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:41.974582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:41.974591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:41.974644 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:41.975288 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.975743 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.976503 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.976683 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.976763 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.977809 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:41.977810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:41.977810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:41.977816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:41.978896 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.979253 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.979254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.979255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.979313 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:41.979515 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:41.979516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:41.979516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:41.979644 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:41.979748 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:41.979855 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 05:12:41.979926 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:12:41.981218 19499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:12:41.982730 19499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:12:41.989213 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.984768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45177 (local address 127.0.0.1:43384)
0924 05:12:41.985439 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:41.985447 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:41.985516 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:41.985759 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:41.985759 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:41.985761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:41.985789 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.985790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.986617 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.986618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.987937 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.987939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.988904 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.988905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.988988 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.989003 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 05:12:41.989078 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:12:41.989138 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":27,"thread_start_us":79,"threads_started":1}
I20260924 05:12:41.989233 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.984966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.0.98.3:45177)
0924 05:12:41.985113 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:41.985117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:41.985492 (+ 375us) server_negotiation.cc:407] Connection header received
0924 05:12:41.985601 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:41.985603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:41.985618 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:41.985667 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:41.985874 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.986515 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.988039 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.988735 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.988815 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.988865 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:41.989111 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 05:12:41.989155 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260924 05:12:41.989449 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.984738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40303 (local address 127.0.0.1:50586)
0924 05:12:41.985128 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:41.985165 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:41.985233 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:41.985410 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:41.985413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:41.985416 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:41.985553 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.985556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.986928 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.986930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.987965 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.987967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.988906 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.988906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.988988 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.989003 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 05:12:41.989071 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:12:41.989138 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":4,"thread_start_us":48,"threads_started":1}
I20260924 05:12:41.989753 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.984938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.0.98.1:40303)
0924 05:12:41.985184 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:41.985186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:41.985218 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:12:41.985293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:41.985295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:41.985309 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:41.985406 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:41.985692 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.986825 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.988043 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.988735 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.988809 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.988866 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:41.989182 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 05:12:41.989228 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":73,"threads_started":1}
I20260924 05:12:41.989835 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.984738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35099 (local address 127.0.0.1:51104)
0924 05:12:41.985305 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:41.985312 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:41.985364 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:41.985615 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:41.985616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:41.985618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:41.985671 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.985674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.988055 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.988055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.988704 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:41.988706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.989619 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.989619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:41.989680 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.989684 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:12:41.989730 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 05:12:41.989766 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260924 05:12:41.989902 19892 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5f0105746e4856816239864545702a (DEFAULT_TABLE table=test-table [id=784230206c0a457fadec2fc23ed690f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:41.990098 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:41.984936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.0.98.2:35099)
0924 05:12:41.985094 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:41.985099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:41.985375 (+ 276us) server_negotiation.cc:407] Connection header received
0924 05:12:41.985426 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:41.985428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:41.985444 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:41.985519 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:41.985741 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.986825 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.988792 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:41.989465 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:41.989533 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:41.989585 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:41.989972 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 05:12:41.990017 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260924 05:12:41.990259 19892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5f0105746e4856816239864545702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:41.990620 19758 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5f0105746e4856816239864545702a (DEFAULT_TABLE table=test-table [id=784230206c0a457fadec2fc23ed690f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:41.990701 19624 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5f0105746e4856816239864545702a (DEFAULT_TABLE table=test-table [id=784230206c0a457fadec2fc23ed690f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:41.990923 19758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5f0105746e4856816239864545702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:41.991075 19624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5f0105746e4856816239864545702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:41.993542 19972 tablet_bootstrap.cc:492] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79: Bootstrap starting.
I20260924 05:12:41.994138 19972 tablet_bootstrap.cc:654] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:41.994227 19974 tablet_bootstrap.cc:492] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4: Bootstrap starting.
I20260924 05:12:41.994313 19973 tablet_bootstrap.cc:492] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: Bootstrap starting.
I20260924 05:12:41.994867 19972 log.cc:824] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:41.994899 19973 tablet_bootstrap.cc:654] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:41.994910 19974 tablet_bootstrap.cc:654] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:41.995436 19973 log.cc:824] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:41.995435 19974 log.cc:824] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:41.995656 19972 tablet_bootstrap.cc:492] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79: No bootstrap required, opened a new log
I20260924 05:12:41.995740 19972 ts_tablet_manager.cc:1397] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:12:41.996227 19973 tablet_bootstrap.cc:492] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: No bootstrap required, opened a new log
I20260924 05:12:41.996307 19973 ts_tablet_manager.cc:1397] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:12:41.996407 19974 tablet_bootstrap.cc:492] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4: No bootstrap required, opened a new log
I20260924 05:12:41.996474 19974 ts_tablet_manager.cc:1397] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:12:41.997633 19973 raft_consensus.cc:348] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } }
I20260924 05:12:41.997788 19973 raft_consensus.cc:374] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:41.997831 19973 raft_consensus.cc:729] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d7b36ea86440aaa4539418aaad2e71, State: Initialized, Role: FOLLOWER
I20260924 05:12:41.997812 19972 raft_consensus.cc:348] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } }
I20260924 05:12:41.997949 19973 consensus_queue.cc:260] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } }
I20260924 05:12:41.998028 19972 raft_consensus.cc:374] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:41.998066 19972 raft_consensus.cc:729] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf0aae1a4294548a6c0bcaca67e3c79, State: Initialized, Role: FOLLOWER
I20260924 05:12:41.998194 19972 consensus_queue.cc:260] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } }
I20260924 05:12:41.998229 19973 ts_tablet_manager.cc:1428] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:12:41.998265 19958 heartbeater.cc:503] Master 127.0.98.62:38289 was elected leader, sending a full tablet report...
I20260924 05:12:41.998461 19972 ts_tablet_manager.cc:1428] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:12:41.998487 19824 heartbeater.cc:503] Master 127.0.98.62:38289 was elected leader, sending a full tablet report...
I20260924 05:12:41.998420 19974 raft_consensus.cc:348] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } }
I20260924 05:12:41.998579 19974 raft_consensus.cc:374] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:41.998605 19974 raft_consensus.cc:729] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d94d09421d04aa3b57eb0a2310ff4c4, State: Initialized, Role: FOLLOWER
I20260924 05:12:41.998708 19974 consensus_queue.cc:260] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } }
I20260924 05:12:41.999019 19974 ts_tablet_manager.cc:1428] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:12:41.999100 19690 heartbeater.cc:503] Master 127.0.98.62:38289 was elected leader, sending a full tablet report...
W20260924 05:12:42.071084 19825 tablet.cc:2369] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:12:42.120555 19978 raft_consensus.cc:482] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:12:42.120682 19978 raft_consensus.cc:504] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } }
I20260924 05:12:42.121054 19978 leader_election.cc:288] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf0aae1a4294548a6c0bcaca67e3c79 (127.0.98.2:35099), 6d94d09421d04aa3b57eb0a2310ff4c4 (127.0.98.1:40303)
I20260924 05:12:42.124962 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.121325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47669 (local address 127.0.98.2:35099)
0924 05:12:42.121371 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.121373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.121385 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:12:42.121416 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.121417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.121426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.121502 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:42.121704 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.122469 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.124005 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.124584 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.124647 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.124656 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.124828 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.124891 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:12:42.124955 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.121234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35099 (local address 127.0.98.3:47669)
0924 05:12:42.121292 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.121303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.121348 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.121530 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.121531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.121533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:42.121605 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.121607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.122568 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.122568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.123913 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.123916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.124662 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.124663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.124744 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.124747 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.124796 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.124854 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":92}
I20260924 05:12:42.124962 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.121335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60163 (local address 127.0.98.1:40303)
0924 05:12:42.121409 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.121411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.121553 (+ 142us) server_negotiation.cc:407] Connection header received
0924 05:12:42.121588 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.121589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.121598 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.121640 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:42.121807 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.122494 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.124004 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.124574 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.124647 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.124656 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.124838 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.124892 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 05:12:42.124967 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.121258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40303 (local address 127.0.98.3:60163)
0924 05:12:42.121461 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.121468 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.121539 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.121669 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.121670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.121671 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:42.121738 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.121739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.122567 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.122568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.123913 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.123916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.124662 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.124663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.124736 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.124739 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.124796 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.124853 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":108,"thread_start_us":95,"threads_started":1}
I20260924 05:12:42.125373 19644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5f0105746e4856816239864545702a" candidate_uuid: "d4d7b36ea86440aaa4539418aaad2e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" is_pre_election: true
I20260924 05:12:42.125427 19776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5f0105746e4856816239864545702a" candidate_uuid: "d4d7b36ea86440aaa4539418aaad2e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" is_pre_election: true
I20260924 05:12:42.125546 19644 raft_consensus.cc:2454] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4d7b36ea86440aaa4539418aaad2e71 in term 0.
I20260924 05:12:42.125566 19776 raft_consensus.cc:2454] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4d7b36ea86440aaa4539418aaad2e71 in term 0.
I20260924 05:12:42.125788 19845 leader_election.cc:302] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf0aae1a4294548a6c0bcaca67e3c79, d4d7b36ea86440aaa4539418aaad2e71; no voters:
I20260924 05:12:42.125927 19978 raft_consensus.cc:2781] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:12:42.125980 19978 raft_consensus.cc:482] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:12:42.126011 19978 raft_consensus.cc:3037] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:42.126667 19978 raft_consensus.cc:504] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } }
I20260924 05:12:42.126832 19978 leader_election.cc:288] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [CANDIDATE]: Term 1 election: Requested vote from peers bbf0aae1a4294548a6c0bcaca67e3c79 (127.0.98.2:35099), 6d94d09421d04aa3b57eb0a2310ff4c4 (127.0.98.1:40303)
I20260924 05:12:42.126957 19776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5f0105746e4856816239864545702a" candidate_uuid: "d4d7b36ea86440aaa4539418aaad2e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79"
I20260924 05:12:42.127020 19644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5f0105746e4856816239864545702a" candidate_uuid: "d4d7b36ea86440aaa4539418aaad2e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4"
I20260924 05:12:42.127056 19776 raft_consensus.cc:3037] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:42.127099 19644 raft_consensus.cc:3037] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:42.127784 19776 raft_consensus.cc:2454] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4d7b36ea86440aaa4539418aaad2e71 in term 1.
I20260924 05:12:42.127910 19644 raft_consensus.cc:2454] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4d7b36ea86440aaa4539418aaad2e71 in term 1.
I20260924 05:12:42.127969 19845 leader_election.cc:302] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf0aae1a4294548a6c0bcaca67e3c79, d4d7b36ea86440aaa4539418aaad2e71; no voters:
I20260924 05:12:42.128042 19978 raft_consensus.cc:2781] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:42.128175 19978 raft_consensus.cc:686] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 1 LEADER]: Becoming Leader. State: Replica: d4d7b36ea86440aaa4539418aaad2e71, State: Running, Role: LEADER
I20260924 05:12:42.128264 19978 consensus_queue.cc:237] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } }
I20260924 05:12:42.129029 19498 catalog_manager.cc:5495] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4d7b36ea86440aaa4539418aaad2e71 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d4d7b36ea86440aaa4539418aaad2e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d7b36ea86440aaa4539418aaad2e71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } health_report { overall_health: HEALTHY } } }
I20260924 05:12:42.168085 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.165673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.0.98.3:45177)
0924 05:12:42.165738 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.165741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.165749 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:12:42.165819 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.165820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.165837 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.165918 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:12:42.166047 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.166500 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.167315 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.167516 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.167598 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.167764 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:12:42.167864 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:12:42.167954 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.167972 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.167991 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:12:42.169600 19644 raft_consensus.cc:1261] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [term 1 FOLLOWER]: Refusing update from remote peer d4d7b36ea86440aaa4539418aaad2e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:42.169632 19776 raft_consensus.cc:1261] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [term 1 FOLLOWER]: Refusing update from remote peer d4d7b36ea86440aaa4539418aaad2e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:42.169899 19982 consensus_queue.cc:1035] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d94d09421d04aa3b57eb0a2310ff4c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:42.170025 19978 consensus_queue.cc:1035] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf0aae1a4294548a6c0bcaca67e3c79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:42.172371 19987 mvcc.cc:205] Tried to move back new op lower bound from 7332768817844572160 to 7332768817678000128. Current Snapshot: MvccSnapshot[applied={T|T < 7332768817844572160}]
I20260924 05:12:42.173019 19986 mvcc.cc:205] Tried to move back new op lower bound from 7332768817844572160 to 7332768817678000128. Current Snapshot: MvccSnapshot[applied={T|T < 7332768817844572160}]
I20260924 05:12:42.174325 19497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:42.174415 19497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:42.175129 19497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45355b28f37e47d0a4952fd370371ce8: Sending DeleteTablet for 3 replicas of tablet 2c5f0105746e4856816239864545702a
I20260924 05:12:42.175426 19758 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c5f0105746e4856816239864545702a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51104
I20260924 05:12:42.175480 19624 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c5f0105746e4856816239864545702a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50586
I20260924 05:12:42.175662 19995 tablet_replica.cc:330] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79: stopping tablet replica
I20260924 05:12:42.175712 19892 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c5f0105746e4856816239864545702a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43384
I20260924 05:12:42.175742 19996 tablet_replica.cc:330] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4: stopping tablet replica
I20260924 05:12:42.175755 19995 raft_consensus.cc:2229] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:12:42.175848 19996 raft_consensus.cc:2229] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:12:42.175865 19995 raft_consensus.cc:2258] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:42.175958 19996 raft_consensus.cc:2258] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:42.176191 19997 tablet_replica.cc:330] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: stopping tablet replica
I20260924 05:12:42.176261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 19561
I20260924 05:12:42.176378 19997 raft_consensus.cc:2229] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:12:42.176473 19996 ts_tablet_manager.cc:1905] T 2c5f0105746e4856816239864545702a P 6d94d09421d04aa3b57eb0a2310ff4c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:42.176508 19997 raft_consensus.cc:2258] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:42.177011 19997 ts_tablet_manager.cc:1905] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:42.178975 19997 ts_tablet_manager.cc:1918] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:42.179047 19997 log.cc:1196] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/2c5f0105746e4856816239864545702a
I20260924 05:12:42.179329 19997 ts_tablet_manager.cc:1939] T 2c5f0105746e4856816239864545702a P d4d7b36ea86440aaa4539418aaad2e71: Deleting consensus metadata
I20260924 05:12:42.182143 19486 catalog_manager.cc:4841] TS d4d7b36ea86440aaa4539418aaad2e71 (127.0.98.3:45177): tablet 2c5f0105746e4856816239864545702a (table test-table [id=784230206c0a457fadec2fc23ed690f0]) successfully deleted
W20260924 05:12:42.182318 19484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40303 (error 108)
I20260924 05:12:42.183009 19995 ts_tablet_manager.cc:1905] T 2c5f0105746e4856816239864545702a P bbf0aae1a4294548a6c0bcaca67e3c79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:42.183952 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.183816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40303 (local address 127.0.0.1:50588)
0924 05:12:42.183877 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.183917 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 05:12:42.183961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 19695
W20260924 05:12:42.184598 19484 catalog_manager.cc:4584] TS 6d94d09421d04aa3b57eb0a2310ff4c4 (127.0.98.1:40303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c5f0105746e4856816239864545702a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40303: connect: Connection refused (error 111)
I20260924 05:12:42.189395 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.189281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35099 (local address 127.0.0.1:51106)
0924 05:12:42.189330 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.189362 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:12:42.189429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 19829
I20260924 05:12:42.193895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 19467
2026-09-24T05:12:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1939 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T05:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:42Z Disabled control of system clock
I20260924 05:12:42.221174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32939
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:12:42.299957 20002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:42.300161 20002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:42.300181 20002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:42.301222 20002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:42.301254 20002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:42.301266 20002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:42.301280 20002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:42.301291 20002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:42.302510 20002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:42.302683 20002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:42.302909 20002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:12:42.307410 20002 server_base.cc:1034] running on GCE node
W20260924 05:12:42.307382 20007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:42.307382 20008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:42.307346 20010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:42.307935 20002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:42.308203 20002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:42.309367 20002 hybrid_clock.cc:648] HybridClock initialized: now 1790226762309358 us; error 41 us; skew 500 ppm
I20260924 05:12:42.310575 20002 webserver.cc:460] Webserver started at http://127.0.98.62:37703/ using document root <none> and password file <none>
I20260924 05:12:42.310880 20002 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:42.310950 20002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:42.311080 20002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:42.312091 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "b2e6958f24fa48e282dcb3ac93f6bf08"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "ba0310a730e42a77a987d8d8f52bdc61"
server_key_iv: "ed6af13971e698e61832175be93da8c7"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.312449 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "b2e6958f24fa48e282dcb3ac93f6bf08"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "ba0310a730e42a77a987d8d8f52bdc61"
server_key_iv: "ed6af13971e698e61832175be93da8c7"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.313674 20002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:12:42.314396 20016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.314576 20002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:42.314689 20002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "b2e6958f24fa48e282dcb3ac93f6bf08"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "ba0310a730e42a77a987d8d8f52bdc61"
server_key_iv: "ed6af13971e698e61832175be93da8c7"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.314788 20002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:42.332806 20002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:42.333092 20002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:42.333253 20002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:42.336903 20002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32939
I20260924 05:12:42.336948 20068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32939 every 8 connection(s)
I20260924 05:12:42.337277 20002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:42.338279 20069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:42.340445 20069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08: Bootstrap starting.
I20260924 05:12:42.340986 20069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:42.341269 20069 log.cc:824] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:42.341884 20069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08: No bootstrap required, opened a new log
I20260924 05:12:42.343493 20069 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6958f24fa48e282dcb3ac93f6bf08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260924 05:12:42.343616 20069 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:42.343662 20069 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e6958f24fa48e282dcb3ac93f6bf08, State: Initialized, Role: FOLLOWER
I20260924 05:12:42.343833 20069 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6958f24fa48e282dcb3ac93f6bf08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260924 05:12:42.343909 20069 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:42.343950 20069 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:42.343997 20069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:42.344485 20069 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6958f24fa48e282dcb3ac93f6bf08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260924 05:12:42.344602 20069 leader_election.cc:302] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2e6958f24fa48e282dcb3ac93f6bf08; no voters:
I20260924 05:12:42.344780 20069 leader_election.cc:288] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:42.344843 20072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:42.345002 20072 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [term 1 LEADER]: Becoming Leader. State: Replica: b2e6958f24fa48e282dcb3ac93f6bf08, State: Running, Role: LEADER
I20260924 05:12:42.345199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 20002
I20260924 05:12:42.345327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:42.345548 392 external_mini_cluster.cc:1428] Setting key 90293a8d1ace005d83adf2f2df01f64b
I20260924 05:12:42.345044 20069 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:42.345712 20072 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6958f24fa48e282dcb3ac93f6bf08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } }
I20260924 05:12:42.346390 20074 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2e6958f24fa48e282dcb3ac93f6bf08. Latest consensus state: current_term: 1 leader_uuid: "b2e6958f24fa48e282dcb3ac93f6bf08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6958f24fa48e282dcb3ac93f6bf08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } } }
I20260924 05:12:42.346544 20074 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:42.347362 20073 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2e6958f24fa48e282dcb3ac93f6bf08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e6958f24fa48e282dcb3ac93f6bf08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32939 } } }
I20260924 05:12:42.347465 20073 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08 [sys.catalog]: This master's current role is: LEADER
W20260924 05:12:42.347724 20090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:12:42.347777 20090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:12:42.347841 20084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:42.347992 20084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:42.349642 20084 catalog_manager.cc:1347] Generated new cluster ID: 346a030ddb684eb992c18f16999b6a5f
I20260924 05:12:42.349699 20084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:42.351480 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.346158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.0.98.62:32939)
0924 05:12:42.346371 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.346379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.346409 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:12:42.346517 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.346524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.346591 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.346721 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:42.347275 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.348475 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.349583 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.350037 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.350178 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.350882 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:42.350886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:42.350886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:42.351063 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:42.351092 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:42.351103 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:42.351177 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.351249 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.351320 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260924 05:12:42.358945 20084 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:42.359405 20084 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:42.366813 20084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08: Generated new TSK 0
I20260924 05:12:42.366997 20084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:42.373121 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:12:42.451313 20093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:42.451520 20093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:42.451552 20093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:42.452665 20093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:42.452724 20093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:42.452766 20093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:12:42.454084 20093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:42.454325 20093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:42.454597 20093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:42.460186 20098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:42.460178 20099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:42.460182 20101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:42.460366 20093 server_base.cc:1034] running on GCE node
I20260924 05:12:42.460590 20093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:42.460862 20093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:42.462044 20093 hybrid_clock.cc:648] HybridClock initialized: now 1790226762462024 us; error 38 us; skew 500 ppm
I20260924 05:12:42.463347 20093 webserver.cc:460] Webserver started at http://127.0.98.1:43429/ using document root <none> and password file <none>
I20260924 05:12:42.463588 20093 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:42.463657 20093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:42.463791 20093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:42.464802 20093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c27e3e5776d4db0967555f4d0bf8fcf"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "e8f2ef115a297d748656af6dadbc4479"
server_key_iv: "90d85680a0c878143030a13bd927cdd4"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.465153 20093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c27e3e5776d4db0967555f4d0bf8fcf"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "e8f2ef115a297d748656af6dadbc4479"
server_key_iv: "90d85680a0c878143030a13bd927cdd4"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.466460 20093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:12:42.467252 20107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.467434 20093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.467530 20093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/wal
uuid: "3c27e3e5776d4db0967555f4d0bf8fcf"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "e8f2ef115a297d748656af6dadbc4479"
server_key_iv: "90d85680a0c878143030a13bd927cdd4"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.467612 20093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:42.489163 20093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:42.489449 20093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:42.489598 20093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:42.489837 20093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:42.490144 20093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:42.490209 20093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.490248 20093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:42.490274 20093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.495805 20093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33735
I20260924 05:12:42.495844 20220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33735 every 8 connection(s)
I20260924 05:12:42.496160 20093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:12:42.497843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 20093
I20260924 05:12:42.497943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:12:42.498180 392 external_mini_cluster.cc:1428] Setting key c2d8c53b7003575eac7c854787966e53
I20260924 05:12:42.499178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:12:42.501660 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.496870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54959 (local address 127.0.98.62:32939)
0924 05:12:42.496933 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.496935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.497160 (+ 225us) server_negotiation.cc:407] Connection header received
0924 05:12:42.497594 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.497594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.497613 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.497694 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:42.498599 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.499164 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.500235 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.500461 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.500521 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.501397 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:42.501398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:42.501399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:42.501403 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:42.501417 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:42.501421 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:42.501472 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.501556 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.501586 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:12:42.501873 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.496863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.1:54959)
0924 05:12:42.497044 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.497076 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.497518 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.497778 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.497783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.497795 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:42.498519 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.498526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.499258 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.499260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.500127 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.500130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.500554 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.500556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.500714 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.501241 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:12:42.501241 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:42.501267 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:12:42.501311 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:42.501483 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:42.501488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:42.501490 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.501557 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.501652 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260924 05:12:42.502384 20221 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32939
I20260924 05:12:42.502485 20221 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:42.502655 20221 heartbeater.cc:511] Master 127.0.98.62:32939 requested a full tablet report, sending...
I20260924 05:12:42.503193 20033 ts_manager.cc:194] Registered new tserver with Master: 3c27e3e5776d4db0967555f4d0bf8fcf (127.0.98.1:33735)
I20260924 05:12:42.504155 20033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54959
W20260924 05:12:42.578271 20224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:42.578460 20224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:42.578480 20224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:42.579623 20224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:42.579661 20224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:42.579685 20224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:12:42.580929 20224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:42.581110 20224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:42.581332 20224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:42.585932 20232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:42.585937 20229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:42.586074 20224 server_base.cc:1034] running on GCE node
W20260924 05:12:42.585945 20230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:42.586393 20224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:42.586617 20224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:42.587779 20224 hybrid_clock.cc:648] HybridClock initialized: now 1790226762587739 us; error 54 us; skew 500 ppm
I20260924 05:12:42.588963 20224 webserver.cc:460] Webserver started at http://127.0.98.2:46757/ using document root <none> and password file <none>
I20260924 05:12:42.589211 20224 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:42.589278 20224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:42.589413 20224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:42.590431 20224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/data/instance:
uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "52c2bd477de6055a5bad352648e6c012"
server_key_iv: "5dd5b82f98c0ccb9f73484467fb583c2"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.590827 20224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "52c2bd477de6055a5bad352648e6c012"
server_key_iv: "5dd5b82f98c0ccb9f73484467fb583c2"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.592089 20224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:12:42.592731 20238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.592887 20224 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:12:42.592979 20224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/wal
uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "52c2bd477de6055a5bad352648e6c012"
server_key_iv: "5dd5b82f98c0ccb9f73484467fb583c2"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.593067 20224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:42.604238 20224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:42.604485 20224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:42.604619 20224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:42.604810 20224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:42.605109 20224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:42.605161 20224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.605201 20224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:42.605225 20224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.610654 20224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40475
I20260924 05:12:42.610687 20351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40475 every 8 connection(s)
I20260924 05:12:42.611019 20224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:12:42.613996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 20224
I20260924 05:12:42.614108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:12:42.614311 392 external_mini_cluster.cc:1428] Setting key 78e8976d57cc2f7071871f0c62ccea38
I20260924 05:12:42.615257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--builtin_ntp_servers=127.0.98.20:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:12:42.616266 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.611704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35877 (local address 127.0.98.62:32939)
0924 05:12:42.611774 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.611777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.611999 (+ 222us) server_negotiation.cc:407] Connection header received
0924 05:12:42.612422 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.612423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.612432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.612483 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:42.613057 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.613597 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.614902 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.615157 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.615210 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.616028 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:42.616029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:42.616029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:42.616033 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:42.616047 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:42.616051 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:42.616095 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.616188 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.616212 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:12:42.616544 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.611677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.2:35877)
0924 05:12:42.611893 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.611922 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.612347 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.612529 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.612531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.612540 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:42.612994 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.612998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.613670 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.613671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.614821 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.614824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.615239 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.615240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.615397 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.615881 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:12:42.615882 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:42.615917 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:12:42.615943 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:42.616113 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:42.616117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:42.616120 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.616192 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.616296 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260924 05:12:42.617014 20352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32939
I20260924 05:12:42.617105 20352 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:42.617259 20352 heartbeater.cc:511] Master 127.0.98.62:32939 requested a full tablet report, sending...
I20260924 05:12:42.617589 20033 ts_manager.cc:194] Registered new tserver with Master: a6cc882b6c9b45f7a5aeee0b49e687c9 (127.0.98.2:40475)
I20260924 05:12:42.618127 20033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35877
W20260924 05:12:42.693569 20355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:42.693738 20355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:42.693758 20355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:42.694818 20355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:42.694850 20355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:42.694875 20355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:12:42.696255 20355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:42.696471 20355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:42.696771 20355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:42.701416 20363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:42.701442 20361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:42.701416 20360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:42.701551 20355 server_base.cc:1034] running on GCE node
I20260924 05:12:42.701750 20355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:42.701993 20355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:42.703154 20355 hybrid_clock.cc:648] HybridClock initialized: now 1790226762703143 us; error 24 us; skew 500 ppm
I20260924 05:12:42.704399 20355 webserver.cc:460] Webserver started at http://127.0.98.3:34501/ using document root <none> and password file <none>
I20260924 05:12:42.704628 20355 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:42.704698 20355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:42.704830 20355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:42.705780 20355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/data/instance:
uuid: "551b00c8b73d40c8b794a4a3c5f04338"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "76c7b3f330318ab20cc00deaeae58cec"
server_key_iv: "d791154672fadaff7a17797998be6349"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.706163 20355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/wal/instance:
uuid: "551b00c8b73d40c8b794a4a3c5f04338"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "76c7b3f330318ab20cc00deaeae58cec"
server_key_iv: "d791154672fadaff7a17797998be6349"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.707460 20355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:12:42.708182 20369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.708334 20355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:42.708451 20355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/wal
uuid: "551b00c8b73d40c8b794a4a3c5f04338"
format_stamp: "Formatted at 2026-09-24 05:12:42 on dist-test-slave-mr2f"
server_key: "76c7b3f330318ab20cc00deaeae58cec"
server_key_iv: "d791154672fadaff7a17797998be6349"
server_key_version: "encryptionkey@0"
I20260924 05:12:42.708518 20355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:42.736832 20355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:42.737136 20355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:42.737293 20355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:42.737528 20355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:12:42.737838 20355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:12:42.737892 20355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.737932 20355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:12:42.737960 20355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:42.743556 20355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35793
I20260924 05:12:42.743597 20482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35793 every 8 connection(s)
I20260924 05:12:42.743894 20355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:12:42.749039 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.744523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42731 (local address 127.0.98.62:32939)
0924 05:12:42.744593 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.744596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.744868 (+ 272us) server_negotiation.cc:407] Connection header received
0924 05:12:42.745418 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.745420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.745446 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.745507 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:42.746215 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.746929 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.747895 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.748073 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.748126 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.748813 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:42.748814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:42.748814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:42.748818 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:42.748831 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:42.748835 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:42.748894 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.748965 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.748985 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:12:42.749191 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.744521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32939 (local address 127.0.98.3:42731)
0924 05:12:42.744715 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.744770 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.745313 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.745575 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.745578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.745587 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:12:42.746132 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.746137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.747006 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.747007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.747820 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.747822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.748139 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.748139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.748238 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.748695 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:12:42.748696 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:12:42.748720 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:12:42.748738 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:12:42.748902 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:12:42.748906 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:12:42.748908 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.748953 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.749018 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260924 05:12:42.749665 20483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32939
I20260924 05:12:42.749752 20483 heartbeater.cc:464] Registering TS with master...
I20260924 05:12:42.749907 20483 heartbeater.cc:511] Master 127.0.98.62:32939 requested a full tablet report, sending...
I20260924 05:12:42.750263 20033 ts_manager.cc:194] Registered new tserver with Master: 551b00c8b73d40c8b794a4a3c5f04338 (127.0.98.3:35793)
I20260924 05:12:42.750571 20033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42731
I20260924 05:12:42.750686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 20355
I20260924 05:12:42.750823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:12:42.751015 392 external_mini_cluster.cc:1428] Setting key 5ced99d91a1ba09826ea27c0c0cfa6c6
I20260924 05:12:42.751783 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:12:42.755528 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.753016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.0.98.62:32939)
0924 05:12:42.753056 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.753058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.753197 (+ 139us) server_negotiation.cc:407] Connection header received
0924 05:12:42.753230 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.753231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.753238 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.753282 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:42.753423 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.753837 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.754528 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.754690 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.754736 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.755314 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:42.755315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:42.755316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:42.755319 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:42.755330 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:42.755333 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:42.755377 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.755421 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.755437 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:12:42.756808 20033 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:12:42.758237 20033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:12:42.764248 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.760232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35793 (local address 127.0.0.1:42516)
0924 05:12:42.760605 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.760613 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.760648 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.760925 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.760927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.760929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:42.760988 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.760990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.761967 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.761968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.763177 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.763180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.763977 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.763978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.764050 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.764063 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.764133 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.764180 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":84,"threads_started":1}
I20260924 05:12:42.764256 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.760653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.0.98.3:35793)
0924 05:12:42.760771 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.760773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.760780 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:12:42.760812 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.760813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.760824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.760889 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:42.761069 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.761884 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.763260 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.763879 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.763972 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.764054 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.764145 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.764180 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260924 05:12:42.764534 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.760226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40475 (local address 127.0.0.1:60648)
0924 05:12:42.760403 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.760439 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.760500 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.760989 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.760990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.760992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:42.761039 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.761041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.762296 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.762297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.763354 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.763355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.764308 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.764309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.764370 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.764375 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.764423 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.764460 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":62,"threads_started":1}
I20260924 05:12:42.764737 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.760258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33735 (local address 127.0.0.1:52382)
0924 05:12:42.761075 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.761084 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.761163 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.761377 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.761378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.761379 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:42.761413 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.761415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.762543 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.762544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.763560 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.763562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.764524 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.764525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.764584 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.764588 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.764641 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.764681 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":64,"threads_started":1}
I20260924 05:12:42.764837 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.760487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.0.98.2:40475)
0924 05:12:42.760807 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.760810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.760818 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:12:42.760854 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.760856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.760888 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.760991 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:42.761126 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.762207 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.763434 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.764122 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.764193 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.764272 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.764697 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.764737 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":82,"threads_started":1}
I20260924 05:12:42.765023 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.760476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.0.98.1:33735)
0924 05:12:42.760618 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.760622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.761136 (+ 514us) server_negotiation.cc:407] Connection header received
0924 05:12:42.761263 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.761265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.761281 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.761360 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:42.761470 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.762464 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.763653 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.764337 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.764414 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.764492 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.764892 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.764942 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260924 05:12:42.765200 20417 tablet_service.cc:1467] Processing CreateTablet for tablet 92747a6ad2514c01b27108c1998a0951 (DEFAULT_TABLE table=test-table [id=eb69b6e24e09486498fe053e79c814ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:42.765527 20417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92747a6ad2514c01b27108c1998a0951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:42.765450 20286 tablet_service.cc:1467] Processing CreateTablet for tablet 92747a6ad2514c01b27108c1998a0951 (DEFAULT_TABLE table=test-table [id=eb69b6e24e09486498fe053e79c814ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:42.765588 20155 tablet_service.cc:1467] Processing CreateTablet for tablet 92747a6ad2514c01b27108c1998a0951 (DEFAULT_TABLE table=test-table [id=eb69b6e24e09486498fe053e79c814ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:12:42.765726 20286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92747a6ad2514c01b27108c1998a0951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:42.765820 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92747a6ad2514c01b27108c1998a0951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:42.768311 20497 tablet_bootstrap.cc:492] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf: Bootstrap starting.
I20260924 05:12:42.768760 20498 tablet_bootstrap.cc:492] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: Bootstrap starting.
I20260924 05:12:42.769101 20499 tablet_bootstrap.cc:492] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: Bootstrap starting.
I20260924 05:12:42.769258 20497 tablet_bootstrap.cc:654] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:42.769393 20498 tablet_bootstrap.cc:654] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:42.769600 20497 log.cc:824] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:42.769714 20499 tablet_bootstrap.cc:654] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:42.769747 20498 log.cc:824] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:42.770061 20499 log.cc:824] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:42.770414 20497 tablet_bootstrap.cc:492] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf: No bootstrap required, opened a new log
I20260924 05:12:42.770449 20498 tablet_bootstrap.cc:492] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: No bootstrap required, opened a new log
I20260924 05:12:42.770478 20497 ts_tablet_manager.cc:1397] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:12:42.770509 20498 ts_tablet_manager.cc:1397] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:12:42.770851 20499 tablet_bootstrap.cc:492] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: No bootstrap required, opened a new log
I20260924 05:12:42.770921 20499 ts_tablet_manager.cc:1397] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:12:42.772392 20497 raft_consensus.cc:348] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } }
I20260924 05:12:42.772536 20497 raft_consensus.cc:374] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:42.772578 20497 raft_consensus.cc:729] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c27e3e5776d4db0967555f4d0bf8fcf, State: Initialized, Role: FOLLOWER
I20260924 05:12:42.772686 20497 consensus_queue.cc:260] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } }
I20260924 05:12:42.772945 20497 ts_tablet_manager.cc:1428] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 05:12:42.773102 20498 raft_consensus.cc:348] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } }
I20260924 05:12:42.773283 20498 raft_consensus.cc:374] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:42.773324 20498 raft_consensus.cc:729] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551b00c8b73d40c8b794a4a3c5f04338, State: Initialized, Role: FOLLOWER
I20260924 05:12:42.773430 20498 consensus_queue.cc:260] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } }
I20260924 05:12:42.773432 20499 raft_consensus.cc:348] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } }
I20260924 05:12:42.773576 20499 raft_consensus.cc:374] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:42.773626 20499 raft_consensus.cc:729] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6cc882b6c9b45f7a5aeee0b49e687c9, State: Initialized, Role: FOLLOWER
I20260924 05:12:42.773692 20498 ts_tablet_manager.cc:1428] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:12:42.773734 20499 consensus_queue.cc:260] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } }
I20260924 05:12:42.773768 20483 heartbeater.cc:503] Master 127.0.98.62:32939 was elected leader, sending a full tablet report...
I20260924 05:12:42.773800 20221 heartbeater.cc:503] Master 127.0.98.62:32939 was elected leader, sending a full tablet report...
I20260924 05:12:42.773999 20499 ts_tablet_manager.cc:1428] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:12:42.774107 20352 heartbeater.cc:503] Master 127.0.98.62:32939 was elected leader, sending a full tablet report...
I20260924 05:12:42.815897 20504 raft_consensus.cc:482] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:12:42.816000 20504 raft_consensus.cc:504] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } }
I20260924 05:12:42.816363 20504 leader_election.cc:288] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6cc882b6c9b45f7a5aeee0b49e687c9 (127.0.98.2:40475), 3c27e3e5776d4db0967555f4d0bf8fcf (127.0.98.1:33735)
I20260924 05:12:42.819875 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.816528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33735 (local address 127.0.98.3:49479)
0924 05:12:42.816732 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.816741 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.816823 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.816946 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.816947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.816948 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:42.817004 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.817005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.817841 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.817842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.819168 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.819170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.819678 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.819679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.819730 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.819734 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.819777 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.819812 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":83,"thread_start_us":111,"threads_started":1}
I20260924 05:12:42.819921 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.816605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49479 (local address 127.0.98.1:33735)
0924 05:12:42.816648 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.816650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.816815 (+ 165us) server_negotiation.cc:407] Connection header received
0924 05:12:42.816880 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.816881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.816890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.816949 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:42.817074 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.817776 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.819241 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.819613 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.819668 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.819675 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.819806 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.819846 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:12:42.820192 20175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92747a6ad2514c01b27108c1998a0951" candidate_uuid: "551b00c8b73d40c8b794a4a3c5f04338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" is_pre_election: true
I20260924 05:12:42.820369 20175 raft_consensus.cc:2454] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 551b00c8b73d40c8b794a4a3c5f04338 in term 0.
I20260924 05:12:42.820508 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.816528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40475 (local address 127.0.98.3:38851)
0924 05:12:42.816603 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.816615 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:12:42.816667 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:12:42.816824 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:12:42.816825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:12:42.816826 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:12:42.816895 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.816896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.817841 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.817842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.819167 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:12:42.819170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.820265 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.820266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:12:42.820339 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.820342 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:12:42.820398 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 05:12:42.820440 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":57}
I20260924 05:12:42.820540 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.816629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38851 (local address 127.0.98.2:40475)
0924 05:12:42.816675 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.816677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.816711 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:12:42.816746 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.816747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.816755 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.816818 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:12:42.816971 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.817751 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.819705 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.820180 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.820263 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.820271 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.820417 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.820463 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:12:42.820598 20370 leader_election.cc:302] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c27e3e5776d4db0967555f4d0bf8fcf, 551b00c8b73d40c8b794a4a3c5f04338; no voters:
I20260924 05:12:42.820766 20504 raft_consensus.cc:2781] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:12:42.820825 20504 raft_consensus.cc:482] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:12:42.820855 20504 raft_consensus.cc:3037] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:42.820832 20306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92747a6ad2514c01b27108c1998a0951" candidate_uuid: "551b00c8b73d40c8b794a4a3c5f04338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" is_pre_election: true
I20260924 05:12:42.820976 20306 raft_consensus.cc:2454] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 551b00c8b73d40c8b794a4a3c5f04338 in term 0.
I20260924 05:12:42.821663 20504 raft_consensus.cc:504] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } }
I20260924 05:12:42.821792 20504 leader_election.cc:288] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [CANDIDATE]: Term 1 election: Requested vote from peers a6cc882b6c9b45f7a5aeee0b49e687c9 (127.0.98.2:40475), 3c27e3e5776d4db0967555f4d0bf8fcf (127.0.98.1:33735)
I20260924 05:12:42.821928 20306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92747a6ad2514c01b27108c1998a0951" candidate_uuid: "551b00c8b73d40c8b794a4a3c5f04338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9"
I20260924 05:12:42.821940 20175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92747a6ad2514c01b27108c1998a0951" candidate_uuid: "551b00c8b73d40c8b794a4a3c5f04338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf"
I20260924 05:12:42.822001 20306 raft_consensus.cc:3037] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:42.822001 20175 raft_consensus.cc:3037] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:42.822906 20306 raft_consensus.cc:2454] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 551b00c8b73d40c8b794a4a3c5f04338 in term 1.
I20260924 05:12:42.822906 20175 raft_consensus.cc:2454] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 551b00c8b73d40c8b794a4a3c5f04338 in term 1.
I20260924 05:12:42.823063 20370 leader_election.cc:302] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c27e3e5776d4db0967555f4d0bf8fcf, 551b00c8b73d40c8b794a4a3c5f04338; no voters:
I20260924 05:12:42.823179 20504 raft_consensus.cc:2781] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:42.823336 20504 raft_consensus.cc:686] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 1 LEADER]: Becoming Leader. State: Replica: 551b00c8b73d40c8b794a4a3c5f04338, State: Running, Role: LEADER
I20260924 05:12:42.823451 20504 consensus_queue.cc:237] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } }
I20260924 05:12:42.824182 20032 catalog_manager.cc:5495] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 reported cstate change: term changed from 0 to 1, leader changed from <none> to 551b00c8b73d40c8b794a4a3c5f04338 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "551b00c8b73d40c8b794a4a3c5f04338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "551b00c8b73d40c8b794a4a3c5f04338" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 } health_report { overall_health: UNKNOWN } } }
I20260924 05:12:42.838045 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.836047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.0.98.3:35793)
0924 05:12:42.836105 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:42.836107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:42.836115 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:12:42.836147 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:42.836148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:42.836170 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:42.836226 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:12:42.836331 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.836753 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.837409 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:42.837576 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:42.837635 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:42.837746 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:12:42.837829 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:12:42.837891 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:42.837923 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 05:12:42.837936 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:12:42.839565 20306 raft_consensus.cc:1261] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [term 1 FOLLOWER]: Refusing update from remote peer 551b00c8b73d40c8b794a4a3c5f04338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:42.839823 20175 raft_consensus.cc:1261] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf [term 1 FOLLOWER]: Refusing update from remote peer 551b00c8b73d40c8b794a4a3c5f04338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:12:42.839862 20504 consensus_queue.cc:1035] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6cc882b6c9b45f7a5aeee0b49e687c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:42.840121 20504 consensus_queue.cc:1035] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c27e3e5776d4db0967555f4d0bf8fcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:12:42.842530 20512 mvcc.cc:205] Tried to move back new op lower bound from 7332768820588703744 to 7332768820525617152. Current Snapshot: MvccSnapshot[applied={T|T < 7332768820588703744}]
I20260924 05:12:42.844214 20032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:42.844297 20032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:12:42.845002 20032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2e6958f24fa48e282dcb3ac93f6bf08: Sending DeleteTablet for 3 replicas of tablet 92747a6ad2514c01b27108c1998a0951
I20260924 05:12:42.845359 20286 tablet_service.cc:1514] Processing DeleteTablet for tablet 92747a6ad2514c01b27108c1998a0951 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:42 UTC) from {username='slave'} at 127.0.0.1:60648
I20260924 05:12:42.845361 20155 tablet_service.cc:1514] Processing DeleteTablet for tablet 92747a6ad2514c01b27108c1998a0951 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:42 UTC) from {username='slave'} at 127.0.0.1:52382
I20260924 05:12:42.845496 20417 tablet_service.cc:1514] Processing DeleteTablet for tablet 92747a6ad2514c01b27108c1998a0951 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:12:42 UTC) from {username='slave'} at 127.0.0.1:42516
I20260924 05:12:42.845654 20520 tablet_replica.cc:330] T 92747a6ad2514c01b27108c1998a0951 P 3c27e3e5776d4db0967555f4d0bf8fcf: stopping tablet replica
I20260924 05:12:42.845736 20521 tablet_replica.cc:330] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: stopping tablet replica
I20260924 05:12:42.845820 20521 raft_consensus.cc:2229] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:12:42.845844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 20093
I20260924 05:12:42.846382 20521 raft_consensus.cc:2258] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:42.846944 20521 ts_tablet_manager.cc:1905] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:42.848258 20522 tablet_replica.cc:330] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: stopping tablet replica
I20260924 05:12:42.848380 20522 raft_consensus.cc:2229] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:12:42.848488 20522 raft_consensus.cc:2258] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:12:42.848698 20521 ts_tablet_manager.cc:1918] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:42.848764 20521 log.cc:1196] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-1/wal/wals/92747a6ad2514c01b27108c1998a0951
I20260924 05:12:42.849000 20522 ts_tablet_manager.cc:1905] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:12:42.849028 20521 ts_tablet_manager.cc:1939] T 92747a6ad2514c01b27108c1998a0951 P a6cc882b6c9b45f7a5aeee0b49e687c9: Deleting consensus metadata
I20260924 05:12:42.849409 20018 catalog_manager.cc:4841] TS a6cc882b6c9b45f7a5aeee0b49e687c9 (127.0.98.2:40475): tablet 92747a6ad2514c01b27108c1998a0951 (table test-table [id=eb69b6e24e09486498fe053e79c814ea]) successfully deleted
I20260924 05:12:42.850437 20522 ts_tablet_manager.cc:1918] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:12:42.850519 20522 log.cc:1196] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: Deleting WAL directory at /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790226527740525-392-0/minicluster-data/ts-2/wal/wals/92747a6ad2514c01b27108c1998a0951
I20260924 05:12:42.850885 20522 ts_tablet_manager.cc:1939] T 92747a6ad2514c01b27108c1998a0951 P 551b00c8b73d40c8b794a4a3c5f04338: Deleting consensus metadata
I20260924 05:12:42.851241 20020 catalog_manager.cc:4841] TS 551b00c8b73d40c8b794a4a3c5f04338 (127.0.98.3:35793): tablet 92747a6ad2514c01b27108c1998a0951 (table test-table [id=eb69b6e24e09486498fe053e79c814ea]) successfully deleted
W20260924 05:12:42.851577 20017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33735 (error 108)
I20260924 05:12:42.851828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 20224
I20260924 05:12:42.852010 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:42.851895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33735 (local address 127.0.0.1:52392)
0924 05:12:42.851940 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:12:42.851975 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 05:12:42.852118 20017 catalog_manager.cc:4584] TS 3c27e3e5776d4db0967555f4d0bf8fcf (127.0.98.1:33735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92747a6ad2514c01b27108c1998a0951: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33735: connect: Connection refused (error 111)
I20260924 05:12:42.856986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 20355
I20260924 05:12:42.861950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 20002
2026-09-24T05:12:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (666 ms)
[----------] 6 tests from EncryptionPolicyTest (7725 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:42 dist-test-slave-mr2f krb5kdc[20526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:42 dist-test-slave-mr2f krb5kdc[20526](info): set up 2 sockets
Sep 24 05:12:42 dist-test-slave-mr2f krb5kdc[20526](info): commencing operation
krb5kdc: starting...
W20260924 05:12:43.878736 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:43 dist-test-slave-mr2f krb5kdc[20526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:43.993921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:44.071556 20539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:44.071749 20539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:44.071770 20539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:44.072808 20539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:44.072842 20539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:44.072856 20539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:44.072868 20539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:44.072880 20539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:44.074216 20539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45123
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:44.074379 20539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:44.074524 20539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:44.079033 20544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:44.079063 20545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:44.079248 20539 server_base.cc:1034] running on GCE node
W20260924 05:12:44.079010 20547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:44.079530 20539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:44.079797 20539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:44.080950 20539 hybrid_clock.cc:648] HybridClock initialized: now 1790226764080945 us; error 35 us; skew 500 ppm
Sep 24 05:12:44 dist-test-slave-mr2f krb5kdc[20526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:44.089025 20539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:44.089442 20539 webserver.cc:460] Webserver started at http://127.0.0.1:44171/ using document root <none> and password file <none>
I20260924 05:12:44.089645 20539 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:44.089709 20539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:44.089851 20539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:44.090874 20539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "dd79d5adcdfe4966aa81106dc535eda5"
format_stamp: "Formatted at 2026-09-24 05:12:44 on dist-test-slave-mr2f"
server_key: "f293c8d39ca87cd81a19a2a5f5f6f9af"
server_key_iv: "23e008647a5b32cf2f6bd64d759880e9"
server_key_version: "encryptionkey@0"
I20260924 05:12:44.091208 20539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd79d5adcdfe4966aa81106dc535eda5"
format_stamp: "Formatted at 2026-09-24 05:12:44 on dist-test-slave-mr2f"
server_key: "f293c8d39ca87cd81a19a2a5f5f6f9af"
server_key_iv: "23e008647a5b32cf2f6bd64d759880e9"
server_key_version: "encryptionkey@0"
I20260924 05:12:44.092439 20539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:12:44.093118 20554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:44.093283 20539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:44.093384 20539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "dd79d5adcdfe4966aa81106dc535eda5"
format_stamp: "Formatted at 2026-09-24 05:12:44 on dist-test-slave-mr2f"
server_key: "f293c8d39ca87cd81a19a2a5f5f6f9af"
server_key_iv: "23e008647a5b32cf2f6bd64d759880e9"
server_key_version: "encryptionkey@0"
I20260924 05:12:44.093454 20539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:44.119040 20539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:44.120859 20539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:44.121002 20539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:44.124897 20539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45123
I20260924 05:12:44.124933 20606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45123 every 8 connection(s)
I20260924 05:12:44.125272 20539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:44.126036 20607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:44.128288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 20539
I20260924 05:12:44.128418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:44.128614 392 external_mini_cluster.cc:1428] Setting key d8b9e2f9b68256f23033888fdfdcd385
I20260924 05:12:44.128885 20607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5: Bootstrap starting.
I20260924 05:12:44.129520 20607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:44.129812 20607 log.cc:824] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:44.130486 20607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5: No bootstrap required, opened a new log
I20260924 05:12:44.131953 20607 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd79d5adcdfe4966aa81106dc535eda5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45123 } }
I20260924 05:12:44.132109 20607 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:44.132160 20607 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd79d5adcdfe4966aa81106dc535eda5, State: Initialized, Role: FOLLOWER
I20260924 05:12:44.132285 20607 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd79d5adcdfe4966aa81106dc535eda5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45123 } }
I20260924 05:12:44.132385 20607 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:44.132427 20607 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:44.132474 20607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:44.132951 20607 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd79d5adcdfe4966aa81106dc535eda5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45123 } }
I20260924 05:12:44.133065 20607 leader_election.cc:302] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd79d5adcdfe4966aa81106dc535eda5; no voters:
I20260924 05:12:44.133267 20607 leader_election.cc:288] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:44.133289 20612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:44.133406 20612 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [term 1 LEADER]: Becoming Leader. State: Replica: dd79d5adcdfe4966aa81106dc535eda5, State: Running, Role: LEADER
I20260924 05:12:44.133487 20612 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd79d5adcdfe4966aa81106dc535eda5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45123 } }
I20260924 05:12:44.133527 20607 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:44.133797 20613 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd79d5adcdfe4966aa81106dc535eda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd79d5adcdfe4966aa81106dc535eda5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45123 } } }
I20260924 05:12:44.133888 20613 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:44.134040 20614 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd79d5adcdfe4966aa81106dc535eda5. Latest consensus state: current_term: 1 leader_uuid: "dd79d5adcdfe4966aa81106dc535eda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd79d5adcdfe4966aa81106dc535eda5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45123 } } }
I20260924 05:12:44.134074 20616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:44.134102 20614 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:44.134531 20616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:12:44 dist-test-slave-mr2f krb5kdc[20526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:12:44.135994 20628 mvcc.cc:205] Tried to move back new op lower bound from 7332768825899167744 to 7332768825891131392. Current Snapshot: MvccSnapshot[applied={T|T < 7332768825899167744}]
I20260924 05:12:44.136443 20616 catalog_manager.cc:1347] Generated new cluster ID: b26dec3f5b76476ba2bdff74ff9c2b1b
I20260924 05:12:44.136493 20616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:44.140111 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:44.129080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51005 (local address 127.0.0.1:45123)
0924 05:12:44.129260 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:44.129278 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:44.129315 (+ 37us) server_negotiation.cc:407] Connection header received
0924 05:12:44.129789 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:44.129796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:44.129841 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:44.130005 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:44.130879 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:44.131761 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:44.132890 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:44.133449 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:44.133826 (+ 377us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:44.136466 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:44.136469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:44.136469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:44.136479 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:44.138411 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:44.138875 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:44.138879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:44.138879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:44.138981 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:44.139222 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:44.139223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:44.139223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:44.139520 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:44.139679 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:44.139802 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 05:12:44.139906 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260924 05:12:44.141388 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:44.149507 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:44.142735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51795 (local address 127.0.0.1:45123)
0924 05:12:44.142794 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:44.142796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:44.142952 (+ 156us) server_negotiation.cc:407] Connection header received
0924 05:12:44.142996 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:44.142997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:44.143006 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:44.143059 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:44.143700 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:44.144378 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:44.145302 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:44.145594 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:44.145648 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:44.146936 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:44.146937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:44.146938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:44.146944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:44.148347 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:44.148735 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:44.148736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:44.148736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:44.148787 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:44.149001 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:44.149002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:44.149002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:44.149112 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:44.149218 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:44.149370 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 05:12:44.149428 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:12:44.150476 20616 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:44.151134 20616 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:44.161177 20616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd79d5adcdfe4966aa81106dc535eda5: Generated new TSK 0
I20260924 05:12:44.161352 20616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:44.164206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 20539
W20260924 05:12:44.168587 20524 connection.cc:535] client connection to 127.0.0.1:45123 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45123): Connection reset by peer (error 104)
2026-09-24T05:12:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:44 dist-test-slave-mr2f krb5kdc[20638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:44 dist-test-slave-mr2f krb5kdc[20638](info): set up 2 sockets
Sep 24 05:12:44 dist-test-slave-mr2f krb5kdc[20638](info): commencing operation
krb5kdc: starting...
W20260924 05:12:45.190745 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:45 dist-test-slave-mr2f krb5kdc[20638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:45.301394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:45.378628 20651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:45.378829 20651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:45.378852 20651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:45.379863 20651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:45.379895 20651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:45.379909 20651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:45.379921 20651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:45.379932 20651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:45.381287 20651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:45.381456 20651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:45.381690 20651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:45.386096 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:45.386190 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:45.386101 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:45.386327 20651 server_base.cc:1034] running on GCE node
I20260924 05:12:45.386554 20651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:45.386891 20651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:45.388036 20651 hybrid_clock.cc:648] HybridClock initialized: now 1790226765388017 us; error 36 us; skew 500 ppm
Sep 24 05:12:45 dist-test-slave-mr2f krb5kdc[20638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:45.399008 20651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:45.399446 20651 webserver.cc:460] Webserver started at http://127.0.0.1:37123/ using document root <none> and password file <none>
I20260924 05:12:45.399683 20651 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:45.399753 20651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:45.399904 20651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:45.400931 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "adb19b35332d43db92b641dd2615894a"
format_stamp: "Formatted at 2026-09-24 05:12:45 on dist-test-slave-mr2f"
server_key: "938ebaf1a5f3d24bb8cb429b9447afd2"
server_key_iv: "d75979a4eb46d6a67e980e468a95eb4f"
server_key_version: "encryptionkey@0"
I20260924 05:12:45.401281 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "adb19b35332d43db92b641dd2615894a"
format_stamp: "Formatted at 2026-09-24 05:12:45 on dist-test-slave-mr2f"
server_key: "938ebaf1a5f3d24bb8cb429b9447afd2"
server_key_iv: "d75979a4eb46d6a67e980e468a95eb4f"
server_key_version: "encryptionkey@0"
I20260924 05:12:45.402554 20651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:12:45.403338 20666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:45.403522 20651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:45.403616 20651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "adb19b35332d43db92b641dd2615894a"
format_stamp: "Formatted at 2026-09-24 05:12:45 on dist-test-slave-mr2f"
server_key: "938ebaf1a5f3d24bb8cb429b9447afd2"
server_key_iv: "d75979a4eb46d6a67e980e468a95eb4f"
server_key_version: "encryptionkey@0"
I20260924 05:12:45.403718 20651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:45.432785 20651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:45.434486 20651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:45.434644 20651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:45.438254 20651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34611
I20260924 05:12:45.438325 20718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34611 every 8 connection(s)
I20260924 05:12:45.438655 20651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:45.439487 20719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:45.441524 20719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a: Bootstrap starting.
I20260924 05:12:45.442052 20719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:45.442345 20719 log.cc:824] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:45.443298 20719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a: No bootstrap required, opened a new log
I20260924 05:12:45.444921 20719 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb19b35332d43db92b641dd2615894a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260924 05:12:45.445072 20719 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:45.445117 20719 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb19b35332d43db92b641dd2615894a, State: Initialized, Role: FOLLOWER
I20260924 05:12:45.445264 20719 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb19b35332d43db92b641dd2615894a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260924 05:12:45.445343 20719 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:45.445382 20719 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:45.445430 20719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:45.445828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 20651
I20260924 05:12:45.445948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:45.445971 20719 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb19b35332d43db92b641dd2615894a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260924 05:12:45.446071 20719 leader_election.cc:302] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb19b35332d43db92b641dd2615894a; no voters:
I20260924 05:12:45.446141 392 external_mini_cluster.cc:1428] Setting key b9a490db8fd9f86192e168b1be6d85f8
I20260924 05:12:45.446233 20719 leader_election.cc:288] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:45.446271 20722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:45.446424 20722 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [term 1 LEADER]: Becoming Leader. State: Replica: adb19b35332d43db92b641dd2615894a, State: Running, Role: LEADER
I20260924 05:12:45.446489 20722 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb19b35332d43db92b641dd2615894a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260924 05:12:45.446507 20719 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:45.446771 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb19b35332d43db92b641dd2615894a. Latest consensus state: current_term: 1 leader_uuid: "adb19b35332d43db92b641dd2615894a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb19b35332d43db92b641dd2615894a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } } }
I20260924 05:12:45.446803 20723 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb19b35332d43db92b641dd2615894a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb19b35332d43db92b641dd2615894a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } } }
I20260924 05:12:45.446835 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:45.446846 20723 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:45.447021 20727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:45.447515 20727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:45.449123 20727 catalog_manager.cc:1347] Generated new cluster ID: 60c577ca6e4948acbad2f34ac0cee944
I20260924 05:12:45.449189 20727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:45.453361 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:45.447071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39215 (local address 127.0.0.1:34611)
0924 05:12:45.448090 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:45.448098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:45.448125 (+ 27us) server_negotiation.cc:407] Connection header received
0924 05:12:45.448262 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:45.448269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:45.448318 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:45.448453 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:45.448796 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:45.450143 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:45.451358 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:45.451934 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:45.452035 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:45.452696 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:45.452700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:45.452700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:45.452884 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:45.452919 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:45.452932 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:45.453010 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:45.453125 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 05:12:45.453151 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":78,"threads_started":1}
I20260924 05:12:45.454264 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:45.460836 20727 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:45.461340 20727 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:45.462473 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:45.459097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42859 (local address 127.0.0.1:34611)
0924 05:12:45.459148 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:45.459151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:45.459208 (+ 57us) server_negotiation.cc:407] Connection header received
0924 05:12:45.459250 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:45.459251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:45.459274 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:45.459342 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:45.459447 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:45.460159 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:45.461098 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:45.461405 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:45.461469 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:45.462122 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:45.462123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:45.462123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:45.462127 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:45.462140 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:45.462144 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:45.462196 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:45.462313 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 05:12:45.462408 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:12:45.468940 20727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb19b35332d43db92b641dd2615894a: Generated new TSK 0
I20260924 05:12:45.469105 20727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:45.471290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 20651
W20260924 05:12:45.475548 20636 connection.cc:535] client connection to 127.0.0.1:34611 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34611): Connection reset by peer (error 104)
2026-09-24T05:12:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:45 dist-test-slave-mr2f krb5kdc[20750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:45 dist-test-slave-mr2f krb5kdc[20750](info): set up 2 sockets
Sep 24 05:12:45 dist-test-slave-mr2f krb5kdc[20750](info): commencing operation
krb5kdc: starting...
W20260924 05:12:46.486827 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:46 dist-test-slave-mr2f krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:46.599500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:46.675921 20763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:46.676110 20763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:46.676131 20763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:46.677166 20763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:46.677201 20763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:46.677213 20763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:46.677227 20763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:46.677238 20763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:46.678553 20763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:46.678725 20763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:46.678956 20763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:46.683382 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:46.683532 20763 server_base.cc:1034] running on GCE node
W20260924 05:12:46.683383 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:46.683450 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:46.683921 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:46.684144 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:46.685317 20763 hybrid_clock.cc:648] HybridClock initialized: now 1790226766685302 us; error 44 us; skew 500 ppm
Sep 24 05:12:46 dist-test-slave-mr2f krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:46.687638 20763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:46.688071 20763 webserver.cc:460] Webserver started at http://127.0.0.1:44299/ using document root <none> and password file <none>
I20260924 05:12:46.688302 20763 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:46.688369 20763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:46.688530 20763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:46.689504 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "22e5d47ea8784cc6a5620915a3cc5260"
format_stamp: "Formatted at 2026-09-24 05:12:46 on dist-test-slave-mr2f"
server_key: "acdb4c384277a822dd7e3c0a64f3ae82"
server_key_iv: "2f6912f575d7a93fdeee04ea13ccfef7"
server_key_version: "encryptionkey@0"
I20260924 05:12:46.689852 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "22e5d47ea8784cc6a5620915a3cc5260"
format_stamp: "Formatted at 2026-09-24 05:12:46 on dist-test-slave-mr2f"
server_key: "acdb4c384277a822dd7e3c0a64f3ae82"
server_key_iv: "2f6912f575d7a93fdeee04ea13ccfef7"
server_key_version: "encryptionkey@0"
I20260924 05:12:46.691180 20763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 05:12:46.691946 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:46.692142 20763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:12:46.692235 20763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "22e5d47ea8784cc6a5620915a3cc5260"
format_stamp: "Formatted at 2026-09-24 05:12:46 on dist-test-slave-mr2f"
server_key: "acdb4c384277a822dd7e3c0a64f3ae82"
server_key_iv: "2f6912f575d7a93fdeee04ea13ccfef7"
server_key_version: "encryptionkey@0"
I20260924 05:12:46.692332 20763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:46.695442 20763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:46.697177 20763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:46.697329 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:46.701036 20763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45045
I20260924 05:12:46.701064 20830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45045 every 8 connection(s)
I20260924 05:12:46.701366 20763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:46.702239 20831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:46.703354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 20763
I20260924 05:12:46.703454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:46.703649 392 external_mini_cluster.cc:1428] Setting key 86f16612685d8208f75416204ed984a8
I20260924 05:12:46.704980 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260: Bootstrap starting.
I20260924 05:12:46.705680 20831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:46.706040 20831 log.cc:824] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:46.706213 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:46.704151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37425 (local address 127.0.0.1:45045)
0924 05:12:46.704346 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:46.704354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:46.704385 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:12:46.704930 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:46.704938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:46.704971 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:46.705116 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:46.705778 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:46.705781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:46.705782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:46.705786 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:46.705829 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:46.705845 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:46.705943 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:46.705975 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 05:12:46.706014 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260924 05:12:46.706727 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260: No bootstrap required, opened a new log
I20260924 05:12:46.707211 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:46.709257 20831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e5d47ea8784cc6a5620915a3cc5260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260924 05:12:46.709431 20831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:46.709467 20831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e5d47ea8784cc6a5620915a3cc5260, State: Initialized, Role: FOLLOWER
I20260924 05:12:46.709578 20831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e5d47ea8784cc6a5620915a3cc5260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260924 05:12:46.709693 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:46.708700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46793 (local address 127.0.0.1:45045)
0924 05:12:46.708737 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:46.708739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:46.708816 (+ 77us) server_negotiation.cc:407] Connection header received
0924 05:12:46.708894 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:46.708894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:46.708897 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:46.708939 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:46.709455 (+ 516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:46.709456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:46.709457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:46.709461 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:46.709475 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:46.709479 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:46.709531 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:46.709619 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 05:12:46.709630 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:12:46.709719 20831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:46.709770 20831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:46.709811 20831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:46.710402 20831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e5d47ea8784cc6a5620915a3cc5260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260924 05:12:46.710484 20831 leader_election.cc:302] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22e5d47ea8784cc6a5620915a3cc5260; no voters:
I20260924 05:12:46.710656 20831 leader_election.cc:288] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:46.710681 20841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:46.710855 20841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [term 1 LEADER]: Becoming Leader. State: Replica: 22e5d47ea8784cc6a5620915a3cc5260, State: Running, Role: LEADER
I20260924 05:12:46.710927 20831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:46.710974 20841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e5d47ea8784cc6a5620915a3cc5260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } }
I20260924 05:12:46.711404 20842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22e5d47ea8784cc6a5620915a3cc5260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e5d47ea8784cc6a5620915a3cc5260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } } }
I20260924 05:12:46.711508 20842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:46.711704 20850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:46.711884 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22e5d47ea8784cc6a5620915a3cc5260. Latest consensus state: current_term: 1 leader_uuid: "22e5d47ea8784cc6a5620915a3cc5260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e5d47ea8784cc6a5620915a3cc5260" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45045 } } }
I20260924 05:12:46.711985 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:46.712193 20850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:46.713663 20850 catalog_manager.cc:1347] Generated new cluster ID: 8ff33f9560f241b0aee2eae732095d30
I20260924 05:12:46.713711 20850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:46.719707 20850 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:46.719866 20850 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:46.724562 20850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22e5d47ea8784cc6a5620915a3cc5260: Generated new TSK 0
I20260924 05:12:46.724764 20850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:46.734326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 20763
2026-09-24T05:12:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1262 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:46 dist-test-slave-mr2f krb5kdc[20862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:46 dist-test-slave-mr2f krb5kdc[20862](info): set up 2 sockets
Sep 24 05:12:46 dist-test-slave-mr2f krb5kdc[20862](info): commencing operation
krb5kdc: starting...
W20260924 05:12:47.742558 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:47 dist-test-slave-mr2f krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:47.853363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:47.932853 20875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:47.933048 20875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:47.933069 20875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:47.934123 20875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:47.934157 20875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:47.934170 20875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:47.934183 20875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:47.934194 20875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:47.935639 20875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:47.935819 20875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:47.935973 20875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:47.940582 20883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:47.940557 20881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:47.940564 20880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:47.940940 20875 server_base.cc:1034] running on GCE node
I20260924 05:12:47.941165 20875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:47.941442 20875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:47.942602 20875 hybrid_clock.cc:648] HybridClock initialized: now 1790226767942577 us; error 35 us; skew 500 ppm
Sep 24 05:12:47 dist-test-slave-mr2f krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:47.944905 20875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:47.945351 20875 webserver.cc:460] Webserver started at http://127.0.0.1:41547/ using document root <none> and password file <none>
I20260924 05:12:47.945562 20875 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:47.945633 20875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:47.945762 20875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:47.946734 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "fc8774f5dbb14050b975eeacf62d45c8"
format_stamp: "Formatted at 2026-09-24 05:12:47 on dist-test-slave-mr2f"
server_key: "ada520ee2737b8471fe1e1d162d75b1d"
server_key_iv: "928589667282b090299f22bf4b9614f3"
server_key_version: "encryptionkey@0"
I20260924 05:12:47.947154 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc8774f5dbb14050b975eeacf62d45c8"
format_stamp: "Formatted at 2026-09-24 05:12:47 on dist-test-slave-mr2f"
server_key: "ada520ee2737b8471fe1e1d162d75b1d"
server_key_iv: "928589667282b090299f22bf4b9614f3"
server_key_version: "encryptionkey@0"
I20260924 05:12:47.948549 20875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:12:47.949247 20890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:47.949416 20875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:47.949509 20875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "fc8774f5dbb14050b975eeacf62d45c8"
format_stamp: "Formatted at 2026-09-24 05:12:47 on dist-test-slave-mr2f"
server_key: "ada520ee2737b8471fe1e1d162d75b1d"
server_key_iv: "928589667282b090299f22bf4b9614f3"
server_key_version: "encryptionkey@0"
I20260924 05:12:47.949590 20875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:47.974233 20875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:47.976158 20875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:47.976325 20875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:47.980314 20875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37443
I20260924 05:12:47.980352 20942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37443 every 8 connection(s)
I20260924 05:12:47.980737 20875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:47.981544 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:47.983868 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8: Bootstrap starting.
I20260924 05:12:47.984462 20943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:47.984760 20943 log.cc:824] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:47.985450 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8: No bootstrap required, opened a new log
I20260924 05:12:47.987018 20943 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8774f5dbb14050b975eeacf62d45c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260924 05:12:47.987152 20943 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:47.987174 20943 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8774f5dbb14050b975eeacf62d45c8, State: Initialized, Role: FOLLOWER
I20260924 05:12:47.987351 20943 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8774f5dbb14050b975eeacf62d45c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260924 05:12:47.987426 20943 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:47.987447 20943 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:47.987511 20943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:47.988039 20943 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8774f5dbb14050b975eeacf62d45c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260924 05:12:47.988119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 20875
I20260924 05:12:47.988148 20943 leader_election.cc:302] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc8774f5dbb14050b975eeacf62d45c8; no voters:
I20260924 05:12:47.988260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:47.988315 20943 leader_election.cc:288] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:47.988394 20946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:47.988468 392 external_mini_cluster.cc:1428] Setting key 878f0ac40d1d926d35cbcbfb48fd7137
I20260924 05:12:47.988555 20946 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [term 1 LEADER]: Becoming Leader. State: Replica: fc8774f5dbb14050b975eeacf62d45c8, State: Running, Role: LEADER
I20260924 05:12:47.988643 20946 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8774f5dbb14050b975eeacf62d45c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260924 05:12:47.988988 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc8774f5dbb14050b975eeacf62d45c8. Latest consensus state: current_term: 1 leader_uuid: "fc8774f5dbb14050b975eeacf62d45c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8774f5dbb14050b975eeacf62d45c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } } }
I20260924 05:12:47.989061 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:47.988634 20943 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:47.989210 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc8774f5dbb14050b975eeacf62d45c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8774f5dbb14050b975eeacf62d45c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } } }
I20260924 05:12:47.989269 20952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:47.989363 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:47.989985 20952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:47.991741 20952 catalog_manager.cc:1347] Generated new cluster ID: 178ee754965f425ea77cbdaa075150a8
I20260924 05:12:47.991791 20952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:12:47 dist-test-slave-mr2f krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:12:48.000620 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:47.988908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34515 (local address 127.0.0.1:37443)
0924 05:12:47.989220 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:47.989243 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:47.989275 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:12:47.989396 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:47.989401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:47.989456 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:47.989621 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:47.990492 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:47.991796 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:47.992876 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:47.993518 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:47.993632 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:47.996383 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:47.996387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:47.996387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:47.996399 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:47.998986 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:47.999390 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:47.999393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:47.999393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:47.999506 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:47.999719 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:47.999720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:47.999720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:48.000009 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:48.000162 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:48.000275 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 05:12:48.000401 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":69,"threads_started":1}
I20260924 05:12:48.001523 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:48.009892 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:48.003006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50585 (local address 127.0.0.1:37443)
0924 05:12:48.003046 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:48.003048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:48.003159 (+ 111us) server_negotiation.cc:407] Connection header received
0924 05:12:48.003187 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:48.003187 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:48.003198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:48.003272 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:48.003846 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:48.004583 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:48.005502 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:48.005813 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:48.005880 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:48.007194 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:48.007196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:48.007196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:48.007203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:48.008789 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:48.009159 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:48.009160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:48.009161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:48.009217 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:48.009417 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:48.009418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:48.009418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:48.009540 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:48.009631 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:48.009730 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 05:12:48.009802 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:12:48.017580 20952 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:48.018123 20952 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:48.024487 20952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc8774f5dbb14050b975eeacf62d45c8: Generated new TSK 0
I20260924 05:12:48.024664 20952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:48.033838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 20875
2026-09-24T05:12:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:48 dist-test-slave-mr2f krb5kdc[20974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:48 dist-test-slave-mr2f krb5kdc[20974](info): set up 2 sockets
Sep 24 05:12:48 dist-test-slave-mr2f krb5kdc[20974](info): commencing operation
krb5kdc: starting...
W20260924 05:12:49.046589 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:49 dist-test-slave-mr2f krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:49.157482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:49.236048 20987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:49.236228 20987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:49.236248 20987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:49.237339 20987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:49.237378 20987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:49.237392 20987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:49.237429 20987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:49.237454 20987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:49.238868 20987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:49.239086 20987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:49.239295 20987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:49.243957 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:49.243968 20995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:49.243968 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:49.243999 20987 server_base.cc:1034] running on GCE node
I20260924 05:12:49.244360 20987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:49.244612 20987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:49.245771 20987 hybrid_clock.cc:648] HybridClock initialized: now 1790226769245755 us; error 28 us; skew 500 ppm
Sep 24 05:12:49 dist-test-slave-mr2f krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:49.248154 20987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:49.248556 20987 webserver.cc:460] Webserver started at http://127.0.0.1:43579/ using document root <none> and password file <none>
I20260924 05:12:49.248816 20987 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:49.248884 20987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:49.249027 20987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:49.250010 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "8d5784f9748141118311f27bbfcdbcb0"
format_stamp: "Formatted at 2026-09-24 05:12:49 on dist-test-slave-mr2f"
server_key: "038fe9c3ee5253ee8c3c59d05bca1aa0"
server_key_iv: "618d5a1e271d59140f3439314e1b5e46"
server_key_version: "encryptionkey@0"
I20260924 05:12:49.250362 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d5784f9748141118311f27bbfcdbcb0"
format_stamp: "Formatted at 2026-09-24 05:12:49 on dist-test-slave-mr2f"
server_key: "038fe9c3ee5253ee8c3c59d05bca1aa0"
server_key_iv: "618d5a1e271d59140f3439314e1b5e46"
server_key_version: "encryptionkey@0"
I20260924 05:12:49.251760 20987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:12:49.252496 21002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:49.252664 20987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:49.252759 20987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "8d5784f9748141118311f27bbfcdbcb0"
format_stamp: "Formatted at 2026-09-24 05:12:49 on dist-test-slave-mr2f"
server_key: "038fe9c3ee5253ee8c3c59d05bca1aa0"
server_key_iv: "618d5a1e271d59140f3439314e1b5e46"
server_key_version: "encryptionkey@0"
I20260924 05:12:49.252838 20987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:49.285238 20987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:49.287582 20987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:49.287755 20987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:49.291523 20987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42213
I20260924 05:12:49.291560 21054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42213 every 8 connection(s)
I20260924 05:12:49.291917 20987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:49.292273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 20987
I20260924 05:12:49.292409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:49.292634 392 external_mini_cluster.cc:1428] Setting key 29a5c3e9c47879c4a61673fa71e0308a
I20260924 05:12:49.292922 21055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:49.296222 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0: Bootstrap starting.
I20260924 05:12:49.296869 21055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:49.297155 21055 log.cc:824] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:49.297919 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0: No bootstrap required, opened a new log
I20260924 05:12:49.298696 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:49.293112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41953 (local address 127.0.0.1:42213)
0924 05:12:49.293312 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:49.293321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:49.293352 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:12:49.294049 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:49.294059 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:49.294119 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:49.294270 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:49.294418 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:49.295782 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:49.296634 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:49.297253 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:49.297371 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:49.298029 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:49.298034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:49.298034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:49.298245 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:49.298286 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:49.298299 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:49.298362 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:49.298431 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 05:12:49.298494 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260924 05:12:49.299716 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:49.300246 21055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5784f9748141118311f27bbfcdbcb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } }
I20260924 05:12:49.300367 21055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:49.300390 21055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5784f9748141118311f27bbfcdbcb0, State: Initialized, Role: FOLLOWER
I20260924 05:12:49.300516 21055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5784f9748141118311f27bbfcdbcb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } }
I20260924 05:12:49.300601 21055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:49.300657 21055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:49.300705 21055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:49.301309 21055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5784f9748141118311f27bbfcdbcb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } }
I20260924 05:12:49.301415 21055 leader_election.cc:302] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d5784f9748141118311f27bbfcdbcb0; no voters:
I20260924 05:12:49.301569 21055 leader_election.cc:288] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:49.301597 21065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:49.301703 21065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [term 1 LEADER]: Becoming Leader. State: Replica: 8d5784f9748141118311f27bbfcdbcb0, State: Running, Role: LEADER
I20260924 05:12:49.301808 21055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:49.301836 21065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5784f9748141118311f27bbfcdbcb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } }
I20260924 05:12:49.302218 21065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d5784f9748141118311f27bbfcdbcb0. Latest consensus state: current_term: 1 leader_uuid: "8d5784f9748141118311f27bbfcdbcb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5784f9748141118311f27bbfcdbcb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } } }
I20260924 05:12:49.302227 21066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d5784f9748141118311f27bbfcdbcb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5784f9748141118311f27bbfcdbcb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } } }
I20260924 05:12:49.302383 21065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:49.302393 21066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:49.302618 21074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:49.303187 21074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:49.304327 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:49.301084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54551 (local address 127.0.0.1:42213)
0924 05:12:49.301118 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:49.301120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:49.301199 (+ 79us) server_negotiation.cc:407] Connection header received
0924 05:12:49.301257 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:49.301258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:49.301269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:49.301329 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:49.301436 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:49.302094 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:49.303118 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:49.303469 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:49.303563 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:49.304106 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:49.304107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:49.304108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:49.304112 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:49.304124 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:49.304128 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:49.304181 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:49.304238 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 05:12:49.304257 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:12:49.304929 21074 catalog_manager.cc:1347] Generated new cluster ID: 68f1cc76cbdf4596b33c42b847582b21
I20260924 05:12:49.304970 21074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:49.316855 21074 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:49.317324 21074 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:49.319607 21074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d5784f9748141118311f27bbfcdbcb0: Generated new TSK 0
I20260924 05:12:49.319749 21074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:49.322872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 20987
2026-09-24T05:12:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:49 dist-test-slave-mr2f krb5kdc[21086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:49 dist-test-slave-mr2f krb5kdc[21086](info): set up 2 sockets
Sep 24 05:12:49 dist-test-slave-mr2f krb5kdc[21086](info): commencing operation
krb5kdc: starting...
W20260924 05:12:50.322651 392 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:50 dist-test-slave-mr2f krb5kdc[21086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:50.434650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:50.513387 21099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:50.513576 21099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:50.513599 21099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:50.514650 21099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:50.514684 21099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:50.514698 21099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:50.514711 21099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:50.514724 21099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:50.516135 21099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:50.516317 21099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:50.516469 21099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:50.520951 21104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:50.520964 21107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:50.521010 21105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:50.521229 21099 server_base.cc:1034] running on GCE node
I20260924 05:12:50.521459 21099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:50.521726 21099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:50.522895 21099 hybrid_clock.cc:648] HybridClock initialized: now 1790226770522875 us; error 33 us; skew 500 ppm
Sep 24 05:12:50 dist-test-slave-mr2f krb5kdc[21086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:50.528949 21099 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:50.529354 21099 webserver.cc:460] Webserver started at http://127.0.0.1:36945/ using document root <none> and password file <none>
I20260924 05:12:50.529578 21099 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:50.529644 21099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:50.529798 21099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:50.530851 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "43d724c5c54e491ca11d053968b89adb"
format_stamp: "Formatted at 2026-09-24 05:12:50 on dist-test-slave-mr2f"
server_key: "b1e0586de9ab45ea63ddfa80ab593a4e"
server_key_iv: "d62212c34cd92ddab3b04d7e4dc8bbb3"
server_key_version: "encryptionkey@0"
I20260924 05:12:50.531204 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "43d724c5c54e491ca11d053968b89adb"
format_stamp: "Formatted at 2026-09-24 05:12:50 on dist-test-slave-mr2f"
server_key: "b1e0586de9ab45ea63ddfa80ab593a4e"
server_key_iv: "d62212c34cd92ddab3b04d7e4dc8bbb3"
server_key_version: "encryptionkey@0"
I20260924 05:12:50.532440 21099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:12:50.533106 21114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:50.533277 21099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:12:50.533375 21099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "43d724c5c54e491ca11d053968b89adb"
format_stamp: "Formatted at 2026-09-24 05:12:50 on dist-test-slave-mr2f"
server_key: "b1e0586de9ab45ea63ddfa80ab593a4e"
server_key_iv: "d62212c34cd92ddab3b04d7e4dc8bbb3"
server_key_version: "encryptionkey@0"
I20260924 05:12:50.533458 21099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:50.536674 21099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:50.538364 21099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:50.538520 21099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:50.542116 21099 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37691
I20260924 05:12:50.542147 21166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37691 every 8 connection(s)
I20260924 05:12:50.542440 21099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:50.543381 21167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:50.545408 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb: Bootstrap starting.
I20260924 05:12:50.545949 21167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:50.546224 21167 log.cc:824] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:50.546924 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb: No bootstrap required, opened a new log
I20260924 05:12:50.548482 21167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d724c5c54e491ca11d053968b89adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37691 } }
I20260924 05:12:50.548666 21167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:50.548714 21167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43d724c5c54e491ca11d053968b89adb, State: Initialized, Role: FOLLOWER
I20260924 05:12:50.548749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 21099
I20260924 05:12:50.548871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:50.548861 21167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d724c5c54e491ca11d053968b89adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37691 } }
I20260924 05:12:50.548964 21167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:50.549001 21167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:50.549049 21167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:50.549080 392 external_mini_cluster.cc:1428] Setting key 9bca7247c3816fc049f7d0aa81731064
I20260924 05:12:50.550158 21167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d724c5c54e491ca11d053968b89adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37691 } }
I20260924 05:12:50.550331 21167 leader_election.cc:302] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43d724c5c54e491ca11d053968b89adb; no voters:
I20260924 05:12:50.550503 21167 leader_election.cc:288] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:50.550567 21172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:50.550767 21167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:50.550800 21172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [term 1 LEADER]: Becoming Leader. State: Replica: 43d724c5c54e491ca11d053968b89adb, State: Running, Role: LEADER
I20260924 05:12:50.550971 21172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d724c5c54e491ca11d053968b89adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37691 } }
I20260924 05:12:50.551167 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:50.549528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46341 (local address 127.0.0.1:37691)
0924 05:12:50.549757 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:50.549769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:50.549808 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:12:50.549932 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:50.549939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:50.549968 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:50.550102 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:50.550671 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:50.550675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:50.550675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:50.550697 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:50.550739 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:50.550753 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:50.550866 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:50.550960 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 05:12:50.551002 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":119,"threads_started":1}
I20260924 05:12:50.551383 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43d724c5c54e491ca11d053968b89adb. Latest consensus state: current_term: 1 leader_uuid: "43d724c5c54e491ca11d053968b89adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d724c5c54e491ca11d053968b89adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37691 } } }
I20260924 05:12:50.551476 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:50.551765 21174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43d724c5c54e491ca11d053968b89adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d724c5c54e491ca11d053968b89adb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37691 } } }
I20260924 05:12:50.551842 21174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:50.552376 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:50.552490 21184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:50.554176 21184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:50.554651 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:50.553592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53579 (local address 127.0.0.1:37691)
0924 05:12:50.553651 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:50.553653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:50.553733 (+ 80us) server_negotiation.cc:407] Connection header received
0924 05:12:50.553799 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:50.553800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:50.553804 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:50.553856 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:50.554423 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:50.554424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:50.554424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:50.554429 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:50.554441 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:50.554445 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:50.554491 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:50.554551 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 05:12:50.554565 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:12:50.555588 21184 catalog_manager.cc:1347] Generated new cluster ID: 722a88487d6342e9a7d4a1f3644b23ee
I20260924 05:12:50.555641 21184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:50.574394 21184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:50.574550 21184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:50.582612 21184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43d724c5c54e491ca11d053968b89adb: Generated new TSK 0
I20260924 05:12:50.582808 21184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:50.586133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 21099
2026-09-24T05:12:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1262 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:50 dist-test-slave-mr2f krb5kdc[21198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:50 dist-test-slave-mr2f krb5kdc[21198](info): set up 2 sockets
Sep 24 05:12:50 dist-test-slave-mr2f krb5kdc[21198](info): commencing operation
krb5kdc: starting...
W20260924 05:12:51.602648 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:51 dist-test-slave-mr2f krb5kdc[21198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:51.713531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:51.792599 21211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:51.792838 21211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:51.792861 21211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:51.794009 21211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:51.794067 21211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:51.794081 21211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:51.794094 21211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:51.794106 21211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:51.795534 21211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:51.795789 21211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:51.796016 21211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:51.800635 21217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:51.800670 21216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:51.800695 21211 server_base.cc:1034] running on GCE node
W20260924 05:12:51.800670 21219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:51.801076 21211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:51.801352 21211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:51.802517 21211 hybrid_clock.cc:648] HybridClock initialized: now 1790226771802503 us; error 26 us; skew 500 ppm
Sep 24 05:12:51 dist-test-slave-mr2f krb5kdc[21198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:51.804870 21211 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:51.805308 21211 webserver.cc:460] Webserver started at http://127.0.0.1:42935/ using document root <none> and password file <none>
I20260924 05:12:51.805518 21211 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:51.805560 21211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:51.805716 21211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:51.806710 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "6106d3c720014286b1885ef02cc1408d"
format_stamp: "Formatted at 2026-09-24 05:12:51 on dist-test-slave-mr2f"
server_key: "791b0bd9f49fca68f41a33efb3e3c28f"
server_key_iv: "e975270b5af39a22ea8128b5676087f6"
server_key_version: "encryptionkey@0"
I20260924 05:12:51.807117 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "6106d3c720014286b1885ef02cc1408d"
format_stamp: "Formatted at 2026-09-24 05:12:51 on dist-test-slave-mr2f"
server_key: "791b0bd9f49fca68f41a33efb3e3c28f"
server_key_iv: "e975270b5af39a22ea8128b5676087f6"
server_key_version: "encryptionkey@0"
I20260924 05:12:51.808351 21211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:12:51.809010 21226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:51.809181 21211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:51.809274 21211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "6106d3c720014286b1885ef02cc1408d"
format_stamp: "Formatted at 2026-09-24 05:12:51 on dist-test-slave-mr2f"
server_key: "791b0bd9f49fca68f41a33efb3e3c28f"
server_key_iv: "e975270b5af39a22ea8128b5676087f6"
server_key_version: "encryptionkey@0"
I20260924 05:12:51.809346 21211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:51.827080 21211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:51.828948 21211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:51.829120 21211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:51.833197 21211 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46729
I20260924 05:12:51.833249 21278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46729 every 8 connection(s)
I20260924 05:12:51.833600 21211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:51.834493 21279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:51.837956 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d: Bootstrap starting.
I20260924 05:12:51.838155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 21211
I20260924 05:12:51.838281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:51.838495 392 external_mini_cluster.cc:1428] Setting key 533121f3deb5e042de3019c599c9e8a5
I20260924 05:12:51.838766 21279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:51.839111 21279 log.cc:824] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:51.839951 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d: No bootstrap required, opened a new log
I20260924 05:12:51.842213 21279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6106d3c720014286b1885ef02cc1408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } }
I20260924 05:12:51.842375 21279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:51.842409 21279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6106d3c720014286b1885ef02cc1408d, State: Initialized, Role: FOLLOWER
I20260924 05:12:51.842542 21279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6106d3c720014286b1885ef02cc1408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } }
I20260924 05:12:51.842617 21279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:51.842648 21279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:51.842701 21279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:51.843346 21279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6106d3c720014286b1885ef02cc1408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } }
I20260924 05:12:51.843432 21279 leader_election.cc:302] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6106d3c720014286b1885ef02cc1408d; no voters:
I20260924 05:12:51.843611 21279 leader_election.cc:288] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:51.843644 21284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:51.843801 21284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [term 1 LEADER]: Becoming Leader. State: Replica: 6106d3c720014286b1885ef02cc1408d, State: Running, Role: LEADER
I20260924 05:12:51.843875 21279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:51.843988 21284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6106d3c720014286b1885ef02cc1408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } }
I20260924 05:12:51.844336 21285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6106d3c720014286b1885ef02cc1408d. Latest consensus state: current_term: 1 leader_uuid: "6106d3c720014286b1885ef02cc1408d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6106d3c720014286b1885ef02cc1408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } } }
I20260924 05:12:51.844363 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6106d3c720014286b1885ef02cc1408d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6106d3c720014286b1885ef02cc1408d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } } }
I20260924 05:12:51.844455 21285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:51.844555 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:51.844954 21293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:51.845475 21293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:51.847043 21293 catalog_manager.cc:1347] Generated new cluster ID: f282995da2b4475c923cc62aadea4c0a
I20260924 05:12:51.847126 21293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:12:51 dist-test-slave-mr2f krb5kdc[21198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:12:51.851784 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:51.839073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:41991 (local address 127.0.0.1:46729)
0924 05:12:51.839460 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:51.839469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:51.839500 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:12:51.840108 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:51.840116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:51.840180 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:51.840316 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:51.841201 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:51.842055 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:51.843158 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:51.843682 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:51.843796 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:51.848244 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:51.848248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:51.848248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:51.848260 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:51.849967 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:51.850367 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:51.850370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:51.850370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:51.850459 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:51.850689 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:51.850690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:51.850691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:51.851021 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:51.851257 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:51.851343 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 05:12:51.851441 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":80,"threads_started":1}
I20260924 05:12:51.852632 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:51.864845 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:51.855087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:53303 (local address 127.0.0.1:46729)
0924 05:12:51.855121 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:51.855123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:51.858921 (+ 3798us) server_negotiation.cc:407] Connection header received
0924 05:12:51.858993 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:51.858994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:51.859004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:51.859057 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:51.859581 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:51.860061 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:51.860764 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:51.860989 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:51.861074 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:51.862228 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:51.862229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:51.862229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:51.862236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:51.863662 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:51.864028 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:51.864029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:51.864030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:51.864082 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:51.864304 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:51.864306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:51.864306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:51.864433 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:51.864544 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:51.864674 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 05:12:51.864741 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:12:51.874959 21293 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:51.875525 21293 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:51.889639 21293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6106d3c720014286b1885ef02cc1408d: Generated new TSK 0
I20260924 05:12:51.889869 21293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:51.892508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 21211
W20260924 05:12:51.896867 21196 connection.cc:535] client connection to 127.0.0.1:46729 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46729): Connection reset by peer (error 104)
2026-09-24T05:12:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:51 dist-test-slave-mr2f krb5kdc[21310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:51 dist-test-slave-mr2f krb5kdc[21310](info): set up 2 sockets
Sep 24 05:12:51 dist-test-slave-mr2f krb5kdc[21310](info): commencing operation
krb5kdc: starting...
W20260924 05:12:52.902724 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:52 dist-test-slave-mr2f krb5kdc[21310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:53.011926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:53.088323 21323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:53.088537 21323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:53.088572 21323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:53.089680 21323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:53.089746 21323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:53.089771 21323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:53.089793 21323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:53.089848 21323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:53.091187 21323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:53.091418 21323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:53.091609 21323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:53.096174 21329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:53.096239 21331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:53.096201 21328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:53.096304 21323 server_base.cc:1034] running on GCE node
I20260924 05:12:53.096541 21323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:53.096763 21323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:53.097921 21323 hybrid_clock.cc:648] HybridClock initialized: now 1790226773097907 us; error 31 us; skew 500 ppm
Sep 24 05:12:53 dist-test-slave-mr2f krb5kdc[21310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:53.100250 21323 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:53.100679 21323 webserver.cc:460] Webserver started at http://127.0.0.1:35071/ using document root <none> and password file <none>
I20260924 05:12:53.100888 21323 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:53.100951 21323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:53.101105 21323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:53.102118 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "1cc7d34bc0f445bca0ee427acd66f472"
format_stamp: "Formatted at 2026-09-24 05:12:53 on dist-test-slave-mr2f"
server_key: "b0beec696ac6e41efdb10167dac83b67"
server_key_iv: "7e5654316d01e81295778ab5f5164a2d"
server_key_version: "encryptionkey@0"
I20260924 05:12:53.102473 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "1cc7d34bc0f445bca0ee427acd66f472"
format_stamp: "Formatted at 2026-09-24 05:12:53 on dist-test-slave-mr2f"
server_key: "b0beec696ac6e41efdb10167dac83b67"
server_key_iv: "7e5654316d01e81295778ab5f5164a2d"
server_key_version: "encryptionkey@0"
I20260924 05:12:53.103806 21323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 05:12:53.104475 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:53.104642 21323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:53.104734 21323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "1cc7d34bc0f445bca0ee427acd66f472"
format_stamp: "Formatted at 2026-09-24 05:12:53 on dist-test-slave-mr2f"
server_key: "b0beec696ac6e41efdb10167dac83b67"
server_key_iv: "7e5654316d01e81295778ab5f5164a2d"
server_key_version: "encryptionkey@0"
I20260924 05:12:53.104815 21323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:53.123399 21323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:53.125021 21323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:53.125165 21323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:53.128808 21323 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44781
I20260924 05:12:53.128839 21390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44781 every 8 connection(s)
I20260924 05:12:53.129146 21323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:53.129904 21391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:53.131951 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472: Bootstrap starting.
I20260924 05:12:53.132441 21391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:53.132695 21391 log.cc:824] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:53.133373 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472: No bootstrap required, opened a new log
I20260924 05:12:53.134918 21391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7d34bc0f445bca0ee427acd66f472" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260924 05:12:53.135052 21391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:53.135073 21391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc7d34bc0f445bca0ee427acd66f472, State: Initialized, Role: FOLLOWER
I20260924 05:12:53.135254 21391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7d34bc0f445bca0ee427acd66f472" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260924 05:12:53.135315 21391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:53.135335 21391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:53.135403 21391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:53.135938 21391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7d34bc0f445bca0ee427acd66f472" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260924 05:12:53.136046 21391 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc7d34bc0f445bca0ee427acd66f472; no voters:
I20260924 05:12:53.136219 21391 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:53.136241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 21323
I20260924 05:12:53.136271 21394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:53.136376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:53.136411 21394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc7d34bc0f445bca0ee427acd66f472, State: Running, Role: LEADER
I20260924 05:12:53.136453 21391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:53.136495 21394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7d34bc0f445bca0ee427acd66f472" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260924 05:12:53.136577 392 external_mini_cluster.cc:1428] Setting key 9a94c64340ecce34d79b2b4df0e2114d
I20260924 05:12:53.136813 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc7d34bc0f445bca0ee427acd66f472. Latest consensus state: current_term: 1 leader_uuid: "1cc7d34bc0f445bca0ee427acd66f472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7d34bc0f445bca0ee427acd66f472" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } } }
I20260924 05:12:53.136886 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:53.136972 21395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc7d34bc0f445bca0ee427acd66f472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc7d34bc0f445bca0ee427acd66f472" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } } }
I20260924 05:12:53.137171 21395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472 [sys.catalog]: This master's current role is: LEADER
W20260924 05:12:53.138257 21411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:12:53.138312 21411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:12:53.138381 21401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:53.138559 21401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:53.140744 21401 catalog_manager.cc:1347] Generated new cluster ID: 2d3a1250c2a442c3bf93bd40d07f0789
I20260924 05:12:53.140796 21401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:53.142186 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:53.137218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:58507 (local address 127.0.0.1:44781)
0924 05:12:53.137420 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:53.137429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:53.137458 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:12:53.137565 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:53.137572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:53.137628 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:53.137796 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:53.138089 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:53.139333 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:53.140138 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:53.140716 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:53.140824 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:53.141530 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:53.141534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:53.141534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:53.141736 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:53.141768 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:53.141781 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:53.141860 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:53.141973 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 05:12:53.141997 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260924 05:12:53.151217 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:53.156070 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:53.152654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:37245 (local address 127.0.0.1:44781)
0924 05:12:53.152695 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:53.152696 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:53.152786 (+ 90us) server_negotiation.cc:407] Connection header received
0924 05:12:53.152827 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:53.152827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:53.152836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:53.152903 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:53.152997 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:53.153752 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:53.154715 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:53.155077 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:53.155146 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:53.155828 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:53.155829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:53.155830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:53.155834 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:53.155848 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:53.155852 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:53.155915 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:53.155990 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 05:12:53.156001 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:12:53.158164 21401 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:53.158885 21401 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:53.163574 21401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cc7d34bc0f445bca0ee427acd66f472: Generated new TSK 0
I20260924 05:12:53.163730 21401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:53.168143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 21323
W20260924 05:12:53.172681 21308 connection.cc:535] client connection to 127.0.0.1:44781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44781): Connection reset by peer (error 104)
2026-09-24T05:12:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:53 dist-test-slave-mr2f krb5kdc[21422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:53 dist-test-slave-mr2f krb5kdc[21422](info): set up 2 sockets
Sep 24 05:12:53 dist-test-slave-mr2f krb5kdc[21422](info): commencing operation
krb5kdc: starting...
W20260924 05:12:54.182752 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:54 dist-test-slave-mr2f krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:54.297781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:54.375471 21435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:54.375684 21435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:54.375705 21435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:54.376843 21435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:54.376883 21435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:54.376897 21435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:54.376947 21435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:54.376965 21435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:54.378360 21435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:54.378600 21435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:54.378836 21435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:54.383311 21443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:54.383325 21441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:54.383457 21435 server_base.cc:1034] running on GCE node
W20260924 05:12:54.383373 21440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:54.383723 21435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:54.384009 21435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:54.385164 21435 hybrid_clock.cc:648] HybridClock initialized: now 1790226774385151 us; error 26 us; skew 500 ppm
Sep 24 05:12:54 dist-test-slave-mr2f krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:54.387537 21435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:54.387979 21435 webserver.cc:460] Webserver started at http://127.0.0.1:41099/ using document root <none> and password file <none>
I20260924 05:12:54.388181 21435 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:54.388243 21435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:54.388376 21435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:54.389354 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "2def1ea69ec04c0c871605724be75d18"
format_stamp: "Formatted at 2026-09-24 05:12:54 on dist-test-slave-mr2f"
server_key: "5788073662faa6d54f53ae759d25145f"
server_key_iv: "3f0f5e216ca267aed9768f2b24fbe5a0"
server_key_version: "encryptionkey@0"
I20260924 05:12:54.389675 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "2def1ea69ec04c0c871605724be75d18"
format_stamp: "Formatted at 2026-09-24 05:12:54 on dist-test-slave-mr2f"
server_key: "5788073662faa6d54f53ae759d25145f"
server_key_iv: "3f0f5e216ca267aed9768f2b24fbe5a0"
server_key_version: "encryptionkey@0"
I20260924 05:12:54.391001 21435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:12:54.391623 21450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:54.391800 21435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:54.391889 21435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "2def1ea69ec04c0c871605724be75d18"
format_stamp: "Formatted at 2026-09-24 05:12:54 on dist-test-slave-mr2f"
server_key: "5788073662faa6d54f53ae759d25145f"
server_key_iv: "3f0f5e216ca267aed9768f2b24fbe5a0"
server_key_version: "encryptionkey@0"
I20260924 05:12:54.391963 21435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:54.395037 21435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:54.396766 21435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:54.396924 21435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:54.400856 21435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35907
I20260924 05:12:54.400882 21502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35907 every 8 connection(s)
I20260924 05:12:54.401188 21435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:54.402031 21503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:54.402745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 21435
I20260924 05:12:54.402928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:54.403113 392 external_mini_cluster.cc:1428] Setting key 7da22d1c48d08cff6579845fb70f3e75
I20260924 05:12:54.405084 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18: Bootstrap starting.
I20260924 05:12:54.405314 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:54.403590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:55697 (local address 127.0.0.1:35907)
0924 05:12:54.403762 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:54.403770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:54.403804 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:12:54.404183 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:54.404188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:54.404217 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:54.404320 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:54.404894 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:54.404896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:54.404897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:54.404900 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:54.404936 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:54.404945 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:54.404996 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:54.405079 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 05:12:54.405130 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260924 05:12:54.405717 21503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:54.406009 21503 log.cc:824] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:54.406097 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:54.406622 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18: No bootstrap required, opened a new log
I20260924 05:12:54.408550 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:54.407540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:38011 (local address 127.0.0.1:35907)
0924 05:12:54.407574 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:54.407576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:54.407677 (+ 101us) server_negotiation.cc:407] Connection header received
0924 05:12:54.407732 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:54.407732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:54.407735 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:54.407774 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:54.408272 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:54.408273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:54.408273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:54.408276 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:54.408285 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:54.408288 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:54.408327 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:54.408385 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 05:12:54.408394 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:12:54.408946 21503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def1ea69ec04c0c871605724be75d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } }
I20260924 05:12:54.409106 21503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:54.409158 21503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2def1ea69ec04c0c871605724be75d18, State: Initialized, Role: FOLLOWER
I20260924 05:12:54.409265 21503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def1ea69ec04c0c871605724be75d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } }
I20260924 05:12:54.409365 21503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:54.409405 21503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:54.409453 21503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:54.409912 21503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def1ea69ec04c0c871605724be75d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } }
I20260924 05:12:54.410041 21503 leader_election.cc:302] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2def1ea69ec04c0c871605724be75d18; no voters:
I20260924 05:12:54.410220 21503 leader_election.cc:288] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:54.410251 21513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:54.410426 21513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [term 1 LEADER]: Becoming Leader. State: Replica: 2def1ea69ec04c0c871605724be75d18, State: Running, Role: LEADER
I20260924 05:12:54.410455 21503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:54.410553 21513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def1ea69ec04c0c871605724be75d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } }
I20260924 05:12:54.410997 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2def1ea69ec04c0c871605724be75d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def1ea69ec04c0c871605724be75d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } } }
I20260924 05:12:54.411012 21515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2def1ea69ec04c0c871605724be75d18. Latest consensus state: current_term: 1 leader_uuid: "2def1ea69ec04c0c871605724be75d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2def1ea69ec04c0c871605724be75d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } } }
I20260924 05:12:54.411121 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:54.411128 21515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:54.411499 21522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:54.411954 21522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:54.413378 21522 catalog_manager.cc:1347] Generated new cluster ID: 8855580a4fbc47528e4f189ef66ef2b1
I20260924 05:12:54.413426 21522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:54.426620 21522 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:54.426783 21522 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:54.456730 21522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2def1ea69ec04c0c871605724be75d18: Generated new TSK 0
I20260924 05:12:54.456935 21522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:54.461558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 21435
2026-09-24T05:12:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:54 dist-test-slave-mr2f krb5kdc[21534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:54 dist-test-slave-mr2f krb5kdc[21534](info): set up 2 sockets
Sep 24 05:12:54 dist-test-slave-mr2f krb5kdc[21534](info): commencing operation
krb5kdc: starting...
W20260924 05:12:55.470410 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:55 dist-test-slave-mr2f krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:55.580817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:55.657248 21547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:55.657469 21547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:55.657507 21547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:55.658617 21547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:55.658686 21547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:55.658712 21547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:55.658743 21547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:55.658785 21547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:55.660182 21547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:55.660432 21547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:55.660662 21547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:55.665179 21555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:55.665293 21553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:55.665297 21547 server_base.cc:1034] running on GCE node
W20260924 05:12:55.665434 21552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:55.665700 21547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:55.665990 21547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:55.667147 21547 hybrid_clock.cc:648] HybridClock initialized: now 1790226775667142 us; error 34 us; skew 500 ppm
Sep 24 05:12:55 dist-test-slave-mr2f krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:55.675549 21547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:55.675990 21547 webserver.cc:460] Webserver started at http://127.0.0.1:37485/ using document root <none> and password file <none>
I20260924 05:12:55.676204 21547 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:55.676271 21547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:55.676416 21547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:55.677372 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "1943c3212c28436183531983fe960a32"
format_stamp: "Formatted at 2026-09-24 05:12:55 on dist-test-slave-mr2f"
server_key: "83aca2b258dbd35b7107d832c90931ae"
server_key_iv: "14a12aa2380797bb5827d8c7a8bc93bb"
server_key_version: "encryptionkey@0"
I20260924 05:12:55.677701 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "1943c3212c28436183531983fe960a32"
format_stamp: "Formatted at 2026-09-24 05:12:55 on dist-test-slave-mr2f"
server_key: "83aca2b258dbd35b7107d832c90931ae"
server_key_iv: "14a12aa2380797bb5827d8c7a8bc93bb"
server_key_version: "encryptionkey@0"
I20260924 05:12:55.679176 21547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:12:55.679824 21562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:55.679975 21547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:55.680066 21547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "1943c3212c28436183531983fe960a32"
format_stamp: "Formatted at 2026-09-24 05:12:55 on dist-test-slave-mr2f"
server_key: "83aca2b258dbd35b7107d832c90931ae"
server_key_iv: "14a12aa2380797bb5827d8c7a8bc93bb"
server_key_version: "encryptionkey@0"
I20260924 05:12:55.680151 21547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:55.686604 21547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:55.688200 21547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:55.688357 21547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:55.692016 21547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42639
I20260924 05:12:55.692044 21614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42639 every 8 connection(s)
I20260924 05:12:55.692346 21547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:55.693113 21615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:55.694729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 21547
I20260924 05:12:55.694895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:55.695101 392 external_mini_cluster.cc:1428] Setting key a986889872f1f9715b2df218e3231b84
I20260924 05:12:55.695410 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32: Bootstrap starting.
I20260924 05:12:55.696106 21615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:55.696424 21615 log.cc:824] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:55.697103 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32: No bootstrap required, opened a new log
I20260924 05:12:55.699306 21615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943c3212c28436183531983fe960a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42639 } }
I20260924 05:12:55.699463 21615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:55.699498 21615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1943c3212c28436183531983fe960a32, State: Initialized, Role: FOLLOWER
I20260924 05:12:55.699608 21615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943c3212c28436183531983fe960a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42639 } }
I20260924 05:12:55.699672 21615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:55.699700 21615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:55.699743 21615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:55.700419 21615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943c3212c28436183531983fe960a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42639 } }
I20260924 05:12:55.700531 21615 leader_election.cc:302] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1943c3212c28436183531983fe960a32; no voters:
I20260924 05:12:55.700727 21615 leader_election.cc:288] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:55.700750 21620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:55.700871 21620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [term 1 LEADER]: Becoming Leader. State: Replica: 1943c3212c28436183531983fe960a32, State: Running, Role: LEADER
I20260924 05:12:55.700995 21615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:55.701133 21620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943c3212c28436183531983fe960a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42639 } }
I20260924 05:12:55.701581 21622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1943c3212c28436183531983fe960a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943c3212c28436183531983fe960a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42639 } } }
I20260924 05:12:55.701592 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1943c3212c28436183531983fe960a32. Latest consensus state: current_term: 1 leader_uuid: "1943c3212c28436183531983fe960a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1943c3212c28436183531983fe960a32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42639 } } }
I20260924 05:12:55.701766 21622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:55.701776 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32 [sys.catalog]: This master's current role is: LEADER
Sep 24 05:12:55 dist-test-slave-mr2f krb5kdc[21534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:12:55.702553 21629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:55.703176 21629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:55.704593 21629 catalog_manager.cc:1347] Generated new cluster ID: f2ab4ccd200142a496b30c96701fe506
I20260924 05:12:55.704643 21629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:55.706395 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:55.695578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:51765 (local address 127.0.0.1:42639)
0924 05:12:55.695776 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:55.695786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:55.695821 (+ 35us) server_negotiation.cc:407] Connection header received
0924 05:12:55.696405 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:55.696413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:55.696474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:55.696625 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:55.697329 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:55.698654 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:55.699433 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:55.700056 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:55.700167 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:55.702867 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:55.702870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:55.702871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:55.702881 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:55.704745 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:55.705153 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:55.705156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:55.705157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:55.705274 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:55.705495 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:55.705496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:55.705496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:55.705802 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:55.705959 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:55.706077 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 05:12:55.706176 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260924 05:12:55.707424 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:55.711786 21629 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:55.712339 21629 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:55.715636 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:55.708684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:59603 (local address 127.0.0.1:42639)
0924 05:12:55.708762 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:55.708763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:55.708801 (+ 38us) server_negotiation.cc:407] Connection header received
0924 05:12:55.708856 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:55.708857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:55.708869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:55.708935 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:55.709480 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:55.710200 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:55.711164 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:55.711482 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:55.711551 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:55.712932 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:55.712933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:55.712934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:55.712939 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:55.714390 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:55.714838 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:55.714839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:55.714840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:55.714891 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:55.715114 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:55.715115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:55.715115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:55.715225 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:55.715325 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:55.715492 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 05:12:55.715549 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 05:12:55.724324 21629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1943c3212c28436183531983fe960a32: Generated new TSK 0
I20260924 05:12:55.724483 21629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:55.727104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 21547
W20260924 05:12:55.730959 21532 connection.cc:535] client connection to 127.0.0.1:42639 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42639): Connection reset by peer (error 104)
2026-09-24T05:12:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:55 dist-test-slave-mr2f krb5kdc[21646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:55 dist-test-slave-mr2f krb5kdc[21646](info): set up 2 sockets
Sep 24 05:12:55 dist-test-slave-mr2f krb5kdc[21646](info): commencing operation
krb5kdc: starting...
W20260924 05:12:56.734928 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:56 dist-test-slave-mr2f krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:56.845470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:56.927302 21659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:56.927518 21659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:56.927553 21659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:56.928648 21659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:56.928710 21659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:56.928732 21659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:56.928752 21659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:56.928781 21659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:56.930143 21659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:56.930404 21659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:56.930616 21659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:56.935163 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:56.935164 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:56.935205 21659 server_base.cc:1034] running on GCE node
W20260924 05:12:56.935189 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:56.935580 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:56.935851 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:56.936985 21659 hybrid_clock.cc:648] HybridClock initialized: now 1790226776936968 us; error 39 us; skew 500 ppm
Sep 24 05:12:56 dist-test-slave-mr2f krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:56.939399 21659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:56.939824 21659 webserver.cc:460] Webserver started at http://127.0.0.1:36257/ using document root <none> and password file <none>
I20260924 05:12:56.940047 21659 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:56.940090 21659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:56.940253 21659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:56.941263 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "db38a9fd70f84c7390a54517a2a3ee8f"
format_stamp: "Formatted at 2026-09-24 05:12:56 on dist-test-slave-mr2f"
server_key: "4f21693bed794d2d52114028cea6f89c"
server_key_iv: "f17a1cbcf9bf645284983db6ec49f802"
server_key_version: "encryptionkey@0"
I20260924 05:12:56.941632 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "db38a9fd70f84c7390a54517a2a3ee8f"
format_stamp: "Formatted at 2026-09-24 05:12:56 on dist-test-slave-mr2f"
server_key: "4f21693bed794d2d52114028cea6f89c"
server_key_iv: "f17a1cbcf9bf645284983db6ec49f802"
server_key_version: "encryptionkey@0"
I20260924 05:12:56.943054 21659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:12:56.943742 21674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:56.943923 21659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:56.944010 21659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "db38a9fd70f84c7390a54517a2a3ee8f"
format_stamp: "Formatted at 2026-09-24 05:12:56 on dist-test-slave-mr2f"
server_key: "4f21693bed794d2d52114028cea6f89c"
server_key_iv: "f17a1cbcf9bf645284983db6ec49f802"
server_key_version: "encryptionkey@0"
I20260924 05:12:56.944094 21659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:56.963747 21659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:56.965549 21659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:56.965703 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:56.969494 21659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44679
I20260924 05:12:56.969542 21726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44679 every 8 connection(s)
I20260924 05:12:56.969909 21659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:56.970882 21727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:56.972978 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f: Bootstrap starting.
I20260924 05:12:56.973548 21727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:56.973829 21727 log.cc:824] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:56.974468 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f: No bootstrap required, opened a new log
I20260924 05:12:56.976068 21727 raft_consensus.cc:348] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38a9fd70f84c7390a54517a2a3ee8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44679 } }
I20260924 05:12:56.976199 21727 raft_consensus.cc:374] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:56.976245 21727 raft_consensus.cc:729] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db38a9fd70f84c7390a54517a2a3ee8f, State: Initialized, Role: FOLLOWER
I20260924 05:12:56.976394 21727 consensus_queue.cc:260] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38a9fd70f84c7390a54517a2a3ee8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44679 } }
I20260924 05:12:56.976456 21727 raft_consensus.cc:388] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:56.976495 21727 raft_consensus.cc:482] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:56.976543 21727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:56.977034 21727 raft_consensus.cc:504] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38a9fd70f84c7390a54517a2a3ee8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44679 } }
I20260924 05:12:56.977141 21727 leader_election.cc:302] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db38a9fd70f84c7390a54517a2a3ee8f; no voters:
I20260924 05:12:56.977310 21727 leader_election.cc:288] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:56.977337 21730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:56.977506 21730 raft_consensus.cc:686] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [term 1 LEADER]: Becoming Leader. State: Replica: db38a9fd70f84c7390a54517a2a3ee8f, State: Running, Role: LEADER
I20260924 05:12:56.977546 21727 sys_catalog.cc:564] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:56.977631 21730 consensus_queue.cc:237] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38a9fd70f84c7390a54517a2a3ee8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44679 } }
I20260924 05:12:56.978008 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader db38a9fd70f84c7390a54517a2a3ee8f. Latest consensus state: current_term: 1 leader_uuid: "db38a9fd70f84c7390a54517a2a3ee8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38a9fd70f84c7390a54517a2a3ee8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44679 } } }
I20260924 05:12:56.978153 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db38a9fd70f84c7390a54517a2a3ee8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db38a9fd70f84c7390a54517a2a3ee8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44679 } } }
I20260924 05:12:56.978171 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:56.978236 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:56.978616 21739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:56.979126 21739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:56.979907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 21659
I20260924 05:12:56.980019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:56.980226 392 external_mini_cluster.cc:1428] Setting key 650b4311c7536707783b6a02e48cd2b6
I20260924 05:12:56.983173 21739 catalog_manager.cc:1347] Generated new cluster ID: 4e5d55c3f81842b9b4949a4a009a978c
I20260924 05:12:56.983230 21739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:56.988970 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:56.980710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:38407 (local address 127.0.0.1:44679)
0924 05:12:56.980882 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:56.980889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:56.980920 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:12:56.981011 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:56.981016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:56.981058 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:56.981164 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:56.981396 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:56.982255 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:56.987088 (+ 4833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:56.987528 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:56.987652 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:56.988355 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:56.988359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:56.988359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:56.988529 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:56.988556 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:56.988567 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:56.988657 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:56.988742 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 05:12:56.988770 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260924 05:12:56.989938 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:56.994728 21739 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:56.995432 21739 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:56.997352 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:56.991251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:41289 (local address 127.0.0.1:44679)
0924 05:12:56.991297 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:56.991299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:56.994551 (+ 3252us) server_negotiation.cc:407] Connection header received
0924 05:12:56.994622 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:56.994623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:56.994635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:56.994694 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:56.994924 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:56.995402 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:56.996124 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:56.996362 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:56.996448 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:56.997107 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:56.997108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:56.997109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:56.997113 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:56.997126 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:56.997130 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:56.997184 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:56.997247 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 05:12:56.997260 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:12:57.003351 21739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db38a9fd70f84c7390a54517a2a3ee8f: Generated new TSK 0
I20260924 05:12:57.003505 21739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:57.010421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 21659
W20260924 05:12:57.014096 21644 connection.cc:535] client connection to 127.0.0.1:44679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44679): Connection reset by peer (error 104)
2026-09-24T05:12:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:57 dist-test-slave-mr2f krb5kdc[21758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:57 dist-test-slave-mr2f krb5kdc[21758](info): set up 2 sockets
Sep 24 05:12:57 dist-test-slave-mr2f krb5kdc[21758](info): commencing operation
krb5kdc: starting...
W20260924 05:12:58.010968 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:58 dist-test-slave-mr2f krb5kdc[21758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:12:58.117534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:58.192708 21771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:58.192921 21771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:58.192953 21771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:58.194110 21771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:58.194187 21771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:58.194212 21771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:58.194239 21771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:58.194271 21771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:58.195711 21771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:58.195972 21771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:58.196187 21771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:58.200846 21776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:58.200860 21779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:58.200896 21777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:58.201042 21771 server_base.cc:1034] running on GCE node
I20260924 05:12:58.201313 21771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:58.201622 21771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:58.202800 21771 hybrid_clock.cc:648] HybridClock initialized: now 1790226778202792 us; error 34 us; skew 500 ppm
Sep 24 05:12:58 dist-test-slave-mr2f krb5kdc[21758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:58.205109 21771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:12:58.205462 21771 webserver.cc:460] Webserver started at http://127.0.0.1:33747/ using document root <none> and password file <none>
I20260924 05:12:58.205653 21771 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:58.205691 21771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:58.205785 21771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:58.206678 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "bf11485df5db4cf8a71464b74a393b80"
format_stamp: "Formatted at 2026-09-24 05:12:58 on dist-test-slave-mr2f"
server_key: "2cdb7d44a005deb49c57b62043bb33aa"
server_key_iv: "7a8e32b3b4b2eab68192a9a9a42869a2"
server_key_version: "encryptionkey@0"
I20260924 05:12:58.207032 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf11485df5db4cf8a71464b74a393b80"
format_stamp: "Formatted at 2026-09-24 05:12:58 on dist-test-slave-mr2f"
server_key: "2cdb7d44a005deb49c57b62043bb33aa"
server_key_iv: "7a8e32b3b4b2eab68192a9a9a42869a2"
server_key_version: "encryptionkey@0"
I20260924 05:12:58.208225 21771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:12:58.208876 21786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:58.209038 21771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:58.209115 21771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "bf11485df5db4cf8a71464b74a393b80"
format_stamp: "Formatted at 2026-09-24 05:12:58 on dist-test-slave-mr2f"
server_key: "2cdb7d44a005deb49c57b62043bb33aa"
server_key_iv: "7a8e32b3b4b2eab68192a9a9a42869a2"
server_key_version: "encryptionkey@0"
I20260924 05:12:58.209172 21771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:58.212359 21771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:58.214607 21771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:58.214751 21771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:58.218374 21771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44163
I20260924 05:12:58.218456 21838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44163 every 8 connection(s)
I20260924 05:12:58.218720 21771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:58.219640 21839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:58.221396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 21771
I20260924 05:12:58.221515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:58.221721 392 external_mini_cluster.cc:1428] Setting key 06f1576e8a2ff49eb67d9c0a69911980
I20260924 05:12:58.221846 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80: Bootstrap starting.
I20260924 05:12:58.222545 21839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:58.222920 21839 log.cc:824] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:58.223645 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80: No bootstrap required, opened a new log
I20260924 05:12:58.224371 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:58.222175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:57929 (local address 127.0.0.1:44163)
0924 05:12:58.222485 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:58.222493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:58.222525 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:12:58.223184 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:58.223199 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:58.223225 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:58.223367 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:58.223969 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:58.223972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:58.223973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:58.223977 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:58.224022 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:58.224033 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:58.224131 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:58.224170 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 05:12:58.224209 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":72,"threads_started":1}
I20260924 05:12:58.225189 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:58.225759 21839 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11485df5db4cf8a71464b74a393b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } }
I20260924 05:12:58.225898 21839 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:58.225941 21839 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf11485df5db4cf8a71464b74a393b80, State: Initialized, Role: FOLLOWER
I20260924 05:12:58.226049 21839 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11485df5db4cf8a71464b74a393b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } }
I20260924 05:12:58.226138 21839 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:58.226179 21839 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:58.226255 21839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:58.227017 21839 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11485df5db4cf8a71464b74a393b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } }
I20260924 05:12:58.227144 21839 leader_election.cc:302] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf11485df5db4cf8a71464b74a393b80; no voters:
I20260924 05:12:58.227326 21839 leader_election.cc:288] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:58.227361 21849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:58.227526 21849 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [term 1 LEADER]: Becoming Leader. State: Replica: bf11485df5db4cf8a71464b74a393b80, State: Running, Role: LEADER
I20260924 05:12:58.227577 21839 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:58.227638 21849 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11485df5db4cf8a71464b74a393b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } }
I20260924 05:12:58.227595 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:58.226473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:36023 (local address 127.0.0.1:44163)
0924 05:12:58.226544 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:58.226545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:58.226602 (+ 57us) server_negotiation.cc:407] Connection header received
0924 05:12:58.226666 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:58.226667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:58.226672 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:58.226729 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:58.227333 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:58.227334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:58.227335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:12:58.227340 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:58.227353 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:12:58.227357 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:58.227425 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:58.227525 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 05:12:58.227538 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 05:12:58.228148 21851 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf11485df5db4cf8a71464b74a393b80. Latest consensus state: current_term: 1 leader_uuid: "bf11485df5db4cf8a71464b74a393b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11485df5db4cf8a71464b74a393b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } } }
I20260924 05:12:58.228174 21850 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf11485df5db4cf8a71464b74a393b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf11485df5db4cf8a71464b74a393b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } } }
I20260924 05:12:58.228248 21851 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:58.228264 21850 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:58.228436 21858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:58.228904 21858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:58.230326 21858 catalog_manager.cc:1347] Generated new cluster ID: afdaf4e56f3d4c5ba698f82bf2b64580
I20260924 05:12:58.230384 21858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:12:58.246007 21858 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:58.246142 21858 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:58.256887 21858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf11485df5db4cf8a71464b74a393b80: Generated new TSK 0
I20260924 05:12:58.257045 21858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:58.261042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 21771
2026-09-24T05:12:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:58 dist-test-slave-mr2f krb5kdc[21870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:58 dist-test-slave-mr2f krb5kdc[21870](info): set up 2 sockets
Sep 24 05:12:58 dist-test-slave-mr2f krb5kdc[21870](info): commencing operation
krb5kdc: starting...
W20260924 05:12:59.262518 392 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:12:59 dist-test-slave-mr2f krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:12:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:12:59.367571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:12:59.442970 21883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:12:59.443145 21883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:12:59.443166 21883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:12:59.444240 21883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:12:59.444273 21883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:12:59.444288 21883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:12:59.444299 21883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:12:59.444312 21883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:12:59.445689 21883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:12:59.445853 21883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:12:59.446017 21883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:12:59.450490 21888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:12:59.450482 21889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:59.450524 21883 server_base.cc:1034] running on GCE node
W20260924 05:12:59.450464 21891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:12:59.450928 21883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:12:59.451165 21883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:12:59.452294 21883 hybrid_clock.cc:648] HybridClock initialized: now 1790226779452282 us; error 30 us; skew 500 ppm
Sep 24 05:12:59 dist-test-slave-mr2f krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:12:59.454607 21883 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:12:59.455072 21883 webserver.cc:460] Webserver started at http://127.0.98.62:45047/ using document root <none> and password file <none>
I20260924 05:12:59.455314 21883 fs_manager.cc:359] Metadata directory not provided
I20260924 05:12:59.455382 21883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:12:59.455529 21883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:12:59.456509 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "f31ba474aa614b03b50ad5603f5edb39"
format_stamp: "Formatted at 2026-09-24 05:12:59 on dist-test-slave-mr2f"
server_key: "4bfe1096646e3a49d50c5cd1df64e523"
server_key_iv: "84a27da5d324a4624b56b39c60054fb1"
server_key_version: "encryptionkey@0"
I20260924 05:12:59.456861 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "f31ba474aa614b03b50ad5603f5edb39"
format_stamp: "Formatted at 2026-09-24 05:12:59 on dist-test-slave-mr2f"
server_key: "4bfe1096646e3a49d50c5cd1df64e523"
server_key_iv: "84a27da5d324a4624b56b39c60054fb1"
server_key_version: "encryptionkey@0"
I20260924 05:12:59.458066 21883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:12:59.458719 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:12:59.458926 21883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:12:59.459019 21883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "f31ba474aa614b03b50ad5603f5edb39"
format_stamp: "Formatted at 2026-09-24 05:12:59 on dist-test-slave-mr2f"
server_key: "4bfe1096646e3a49d50c5cd1df64e523"
server_key_iv: "84a27da5d324a4624b56b39c60054fb1"
server_key_version: "encryptionkey@0"
I20260924 05:12:59.459105 21883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:12:59.480522 21883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:12:59.482214 21883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:12:59.482373 21883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:12:59.486019 21883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46741
I20260924 05:12:59.486054 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46741 every 8 connection(s)
I20260924 05:12:59.486373 21883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:12:59.487269 21951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:12:59.489297 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39: Bootstrap starting.
I20260924 05:12:59.489773 21951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39: Neither blocks nor log segments found. Creating new log.
I20260924 05:12:59.489984 21951 log.cc:824] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39: Log is configured to *not* fsync() on all Append() calls
I20260924 05:12:59.490594 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39: No bootstrap required, opened a new log
I20260924 05:12:59.491465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 21883
I20260924 05:12:59.491598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:12:59.491778 392 external_mini_cluster.cc:1428] Setting key 61d43abc4e441063ff2676fbf54ecf09
I20260924 05:12:59.492501 21951 raft_consensus.cc:348] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31ba474aa614b03b50ad5603f5edb39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } }
I20260924 05:12:59.492630 21951 raft_consensus.cc:374] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:12:59.492651 21951 raft_consensus.cc:729] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f31ba474aa614b03b50ad5603f5edb39, State: Initialized, Role: FOLLOWER
I20260924 05:12:59.492774 21951 consensus_queue.cc:260] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31ba474aa614b03b50ad5603f5edb39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } }
I20260924 05:12:59.492858 21951 raft_consensus.cc:388] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:12:59.492930 21951 raft_consensus.cc:482] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:12:59.492985 21951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:12:59.493742 21951 raft_consensus.cc:504] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31ba474aa614b03b50ad5603f5edb39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } }
I20260924 05:12:59.493830 21951 leader_election.cc:302] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f31ba474aa614b03b50ad5603f5edb39; no voters:
I20260924 05:12:59.494031 21951 leader_election.cc:288] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:12:59.494061 21956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:12:59.494169 21956 raft_consensus.cc:686] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [term 1 LEADER]: Becoming Leader. State: Replica: f31ba474aa614b03b50ad5603f5edb39, State: Running, Role: LEADER
I20260924 05:12:59.494290 21951 sys_catalog.cc:564] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:12:59.494299 21956 consensus_queue.cc:237] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31ba474aa614b03b50ad5603f5edb39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } }
I20260924 05:12:59.494711 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f31ba474aa614b03b50ad5603f5edb39. Latest consensus state: current_term: 1 leader_uuid: "f31ba474aa614b03b50ad5603f5edb39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31ba474aa614b03b50ad5603f5edb39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } } }
I20260924 05:12:59.494831 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:59.495223 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f31ba474aa614b03b50ad5603f5edb39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31ba474aa614b03b50ad5603f5edb39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } } }
I20260924 05:12:59.495290 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39 [sys.catalog]: This master's current role is: LEADER
I20260924 05:12:59.495527 21963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:12:59.496037 21963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:12:59.497498 21963 catalog_manager.cc:1347] Generated new cluster ID: f586c7ff679b47dda9f9bf462841354a
I20260924 05:12:59.497547 21963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:12:59 dist-test-slave-mr2f krb5kdc[21870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:12:59.502344 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:59.492255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41293 (local address 127.0.98.62:46741)
0924 05:12:59.492443 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:59.492450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:59.492473 (+ 23us) server_negotiation.cc:407] Connection header received
0924 05:12:59.492560 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:59.492565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:59.492602 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:59.492755 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:59.493593 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:59.494759 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:59.495920 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:59.496540 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:59.496674 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:59.499100 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:59.499106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:59.499106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:59.499121 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:59.500793 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:59.501199 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:59.501209 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:59.501209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:59.501303 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:59.501539 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:59.501540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:59.501540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:59.501776 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:59.501957 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:59.502047 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 05:12:59.502149 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260924 05:12:59.503379 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:12:59.512627 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:12:59.504641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46479 (local address 127.0.98.62:46741)
0924 05:12:59.504677 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:12:59.504678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:12:59.506933 (+ 2255us) server_negotiation.cc:407] Connection header received
0924 05:12:59.507003 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:12:59.507005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:12:59.507014 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:12:59.507056 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:12:59.507557 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:59.508017 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:59.508676 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:12:59.508900 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:12:59.508982 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:12:59.510081 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:12:59.510082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:12:59.510083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:12:59.510090 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:12:59.511439 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:59.511841 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:59.511842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:59.511842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:59.511894 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:12:59.512121 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:12:59.512122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:12:59.512122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:12:59.512228 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:12:59.512333 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 05:12:59.512456 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 05:12:59.512518 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:12:59.513990 21963 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:12:59.514647 21963 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:12:59.518087 21963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f31ba474aa614b03b50ad5603f5edb39: Generated new TSK 0
I20260924 05:12:59.518234 21963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:12:59.519896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 21883
W20260924 05:12:59.523506 21868 connection.cc:535] client connection to 127.0.98.62:46741 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46741): Connection reset by peer (error 104)
2026-09-24T05:12:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:12:59 dist-test-slave-mr2f krb5kdc[21982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:12:59 dist-test-slave-mr2f krb5kdc[21982](info): set up 2 sockets
Sep 24 05:12:59 dist-test-slave-mr2f krb5kdc[21982](info): commencing operation
krb5kdc: starting...
W20260924 05:13:00.546943 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:13:00 dist-test-slave-mr2f krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:13:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:13:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:13:00.655102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:13:00.732214 21995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:00.732394 21995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:00.732415 21995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:00.733533 21995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:00.733575 21995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:00.733589 21995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:00.733626 21995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:00.733657 21995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:00.735095 21995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:00.735324 21995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:00.735584 21995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:13:00.740430 21995 server_base.cc:1034] running on GCE node
W20260924 05:13:00.740425 22000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:00.740391 22001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:00.740398 22003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:00.740859 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:00.741127 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:00.742286 21995 hybrid_clock.cc:648] HybridClock initialized: now 1790226780742268 us; error 30 us; skew 500 ppm
Sep 24 05:13:00 dist-test-slave-mr2f krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:00.751185 21995 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:13:00.751652 21995 webserver.cc:460] Webserver started at http://127.0.98.62:39275/ using document root <none> and password file <none>
I20260924 05:13:00.751941 21995 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:00.752008 21995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:00.752154 21995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:00.753176 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "04beb34a5b28416a8a22c3c75b4086bb"
format_stamp: "Formatted at 2026-09-24 05:13:00 on dist-test-slave-mr2f"
server_key: "8d03a2d637a3bc774bc4bc1371c1cbf3"
server_key_iv: "b954aec0664bb52190eb14541bcd1155"
server_key_version: "encryptionkey@0"
I20260924 05:13:00.753527 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "04beb34a5b28416a8a22c3c75b4086bb"
format_stamp: "Formatted at 2026-09-24 05:13:00 on dist-test-slave-mr2f"
server_key: "8d03a2d637a3bc774bc4bc1371c1cbf3"
server_key_iv: "b954aec0664bb52190eb14541bcd1155"
server_key_version: "encryptionkey@0"
I20260924 05:13:00.754860 21995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:13:00.755611 22010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:00.755769 21995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:13:00.755872 21995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "04beb34a5b28416a8a22c3c75b4086bb"
format_stamp: "Formatted at 2026-09-24 05:13:00 on dist-test-slave-mr2f"
server_key: "8d03a2d637a3bc774bc4bc1371c1cbf3"
server_key_iv: "b954aec0664bb52190eb14541bcd1155"
server_key_version: "encryptionkey@0"
I20260924 05:13:00.755959 21995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:00.770054 21995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:00.771860 21995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:00.772014 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:00.775939 21995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39825
I20260924 05:13:00.775969 22062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39825 every 8 connection(s)
I20260924 05:13:00.776298 21995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:13:00.777096 22063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:00.779220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 21995
I20260924 05:13:00.779286 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb: Bootstrap starting.
I20260924 05:13:00.779350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:13:00.779528 392 external_mini_cluster.cc:1428] Setting key a72988fc1d89965d61ee96395bebe1d9
I20260924 05:13:00.779974 22063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:00.780345 22063 log.cc:824] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:00.781064 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb: No bootstrap required, opened a new log
I20260924 05:13:00.783345 22063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04beb34a5b28416a8a22c3c75b4086bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } }
I20260924 05:13:00.783512 22063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:13:00.783547 22063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04beb34a5b28416a8a22c3c75b4086bb, State: Initialized, Role: FOLLOWER
I20260924 05:13:00.783665 22063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04beb34a5b28416a8a22c3c75b4086bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } }
I20260924 05:13:00.783735 22063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:13:00.783763 22063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:13:00.783806 22063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:00.784480 22063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04beb34a5b28416a8a22c3c75b4086bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } }
I20260924 05:13:00.784590 22063 leader_election.cc:302] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04beb34a5b28416a8a22c3c75b4086bb; no voters:
I20260924 05:13:00.784749 22063 leader_election.cc:288] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:13:00.784775 22068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:13:00.784925 22068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [term 1 LEADER]: Becoming Leader. State: Replica: 04beb34a5b28416a8a22c3c75b4086bb, State: Running, Role: LEADER
I20260924 05:13:00.785048 22063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:00.785470 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:00.780030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48113 (local address 127.0.98.62:39825)
0924 05:13:00.780227 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:00.780236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:00.780275 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:13:00.780846 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:00.780854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:00.780915 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:00.781060 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:00.781283 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:00.782663 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:00.783478 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:00.784074 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:00.784193 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:00.784824 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:00.784828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:00.784828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:13:00.785024 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:00.785059 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:13:00.785075 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:00.785197 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:00.785282 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 05:13:00.785304 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260924 05:13:00.786373 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:13:00.785046 22068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04beb34a5b28416a8a22c3c75b4086bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } }
I20260924 05:13:00.786926 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04beb34a5b28416a8a22c3c75b4086bb. Latest consensus state: current_term: 1 leader_uuid: "04beb34a5b28416a8a22c3c75b4086bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04beb34a5b28416a8a22c3c75b4086bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } } }
I20260924 05:13:00.786933 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04beb34a5b28416a8a22c3c75b4086bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04beb34a5b28416a8a22c3c75b4086bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39825 } } }
I20260924 05:13:00.787086 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:00.787101 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb [sys.catalog]: This master's current role is: LEADER
W20260924 05:13:00.787760 22084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:13:00.787814 22084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:13:00.787864 22083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:13:00.788024 22083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:13:00.789283 22083 catalog_manager.cc:1347] Generated new cluster ID: 1f1d7b9f23624a8a8669e5e4566831de
I20260924 05:13:00.789335 22083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:13:00.794243 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:00.788035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46185 (local address 127.0.98.62:39825)
0924 05:13:00.788083 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:00.788085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:00.790956 (+ 2871us) server_negotiation.cc:407] Connection header received
0924 05:13:00.791013 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:00.791014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:00.791023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:00.791077 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:00.791203 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:00.791891 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:00.792889 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:00.793207 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:00.793262 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:00.794002 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:00.794004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:00.794004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:13:00.794008 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:00.794021 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:13:00.794025 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:00.794078 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:00.794165 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:13:00.794174 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:13:00.798720 22083 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:13:00.799180 22083 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:13:00.803356 22083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04beb34a5b28416a8a22c3c75b4086bb: Generated new TSK 0
I20260924 05:13:00.803524 22083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:13:00.812528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 21995
W20260924 05:13:00.816237 21980 connection.cc:535] client connection to 127.0.98.62:39825 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39825): Connection reset by peer (error 104)
2026-09-24T05:13:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:13:00 dist-test-slave-mr2f krb5kdc[22094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:13:00 dist-test-slave-mr2f krb5kdc[22094](info): set up 2 sockets
Sep 24 05:13:00 dist-test-slave-mr2f krb5kdc[22094](info): commencing operation
krb5kdc: starting...
W20260924 05:13:01.826967 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:13:01 dist-test-slave-mr2f krb5kdc[22094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:13:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:13:01.939182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46229
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:13:02.018828 22107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:02.019060 22107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:02.019094 22107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:02.020177 22107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:02.020251 22107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:02.020274 22107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:02.020301 22107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:02.020332 22107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:02.021747 22107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:02.022023 22107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:02.022233 22107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:13:02.026713 22112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:02.026737 22107 server_base.cc:1034] running on GCE node
W20260924 05:13:02.026890 22113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:02.027045 22115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:02.027313 22107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:02.027590 22107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:02.028761 22107 hybrid_clock.cc:648] HybridClock initialized: now 1790226782028744 us; error 29 us; skew 500 ppm
Sep 24 05:13:02 dist-test-slave-mr2f krb5kdc[22094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:02.031113 22107 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:13:02.031499 22107 webserver.cc:460] Webserver started at http://127.0.98.62:35449/ using document root <none> and password file <none>
I20260924 05:13:02.031723 22107 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:02.031765 22107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:02.031899 22107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:02.032857 22107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "2979ad4ab2f843fd928dd7aaed3aeb92"
format_stamp: "Formatted at 2026-09-24 05:13:02 on dist-test-slave-mr2f"
server_key: "98f04929b51fa7417bd140a56ff50f9e"
server_key_iv: "647f0f96e4dd78b743fc17d4b4c95790"
server_key_version: "encryptionkey@0"
I20260924 05:13:02.033186 22107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "2979ad4ab2f843fd928dd7aaed3aeb92"
format_stamp: "Formatted at 2026-09-24 05:13:02 on dist-test-slave-mr2f"
server_key: "98f04929b51fa7417bd140a56ff50f9e"
server_key_iv: "647f0f96e4dd78b743fc17d4b4c95790"
server_key_version: "encryptionkey@0"
I20260924 05:13:02.034364 22107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 05:13:02.035102 22122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:02.035290 22107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:13:02.035386 22107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "2979ad4ab2f843fd928dd7aaed3aeb92"
format_stamp: "Formatted at 2026-09-24 05:13:02 on dist-test-slave-mr2f"
server_key: "98f04929b51fa7417bd140a56ff50f9e"
server_key_iv: "647f0f96e4dd78b743fc17d4b4c95790"
server_key_version: "encryptionkey@0"
I20260924 05:13:02.035478 22107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:02.038468 22107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:02.040189 22107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:02.040339 22107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:02.043979 22107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46229
I20260924 05:13:02.044010 22174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46229 every 8 connection(s)
I20260924 05:13:02.044308 22107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:13:02.045222 22175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:02.047246 22175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92: Bootstrap starting.
I20260924 05:13:02.047715 22175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:02.047935 22175 log.cc:824] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:02.048553 22175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92: No bootstrap required, opened a new log
I20260924 05:13:02.049983 22175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979ad4ab2f843fd928dd7aaed3aeb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46229 } }
I20260924 05:13:02.050125 22175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:13:02.050148 22175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2979ad4ab2f843fd928dd7aaed3aeb92, State: Initialized, Role: FOLLOWER
I20260924 05:13:02.050264 22175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979ad4ab2f843fd928dd7aaed3aeb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46229 } }
I20260924 05:13:02.050384 22175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:13:02.050420 22175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:13:02.050472 22175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:02.051229 22175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979ad4ab2f843fd928dd7aaed3aeb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46229 } }
I20260924 05:13:02.051354 22175 leader_election.cc:302] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2979ad4ab2f843fd928dd7aaed3aeb92; no voters:
I20260924 05:13:02.051514 22175 leader_election.cc:288] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:13:02.051576 22178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:13:02.051754 22178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [term 1 LEADER]: Becoming Leader. State: Replica: 2979ad4ab2f843fd928dd7aaed3aeb92, State: Running, Role: LEADER
I20260924 05:13:02.051759 22175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:02.051846 22178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979ad4ab2f843fd928dd7aaed3aeb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46229 } }
I20260924 05:13:02.052192 22180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2979ad4ab2f843fd928dd7aaed3aeb92. Latest consensus state: current_term: 1 leader_uuid: "2979ad4ab2f843fd928dd7aaed3aeb92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979ad4ab2f843fd928dd7aaed3aeb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46229 } } }
I20260924 05:13:02.052306 22180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:02.052188 22179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2979ad4ab2f843fd928dd7aaed3aeb92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2979ad4ab2f843fd928dd7aaed3aeb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46229 } } }
I20260924 05:13:02.052493 22182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:13:02.052491 22179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:02.053023 22182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:13:02.053397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 22107
I20260924 05:13:02.053514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:13:02.053691 392 external_mini_cluster.cc:1428] Setting key b2da63039f358d6b51fb6a8f45df25b4
I20260924 05:13:02.055177 22182 catalog_manager.cc:1347] Generated new cluster ID: 7611b0563bd44c0297332631d9f7a1b5
I20260924 05:13:02.055226 22182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:13:02.056102 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:02.054333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35577 (local address 127.0.98.62:46229)
0924 05:13:02.054543 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:02.054551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:02.054585 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:13:02.054688 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:02.054694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:02.054731 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:02.054889 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:02.055539 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:02.055543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:02.055543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:13:02.055547 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:02.055586 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:13:02.055600 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:02.055847 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:02.055878 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 05:13:02.055917 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260924 05:13:02.056988 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:13:02.059450 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:02.058361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58555 (local address 127.0.98.62:46229)
0924 05:13:02.058397 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:02.058399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:02.058463 (+ 64us) server_negotiation.cc:407] Connection header received
0924 05:13:02.058552 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:02.058552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:02.058556 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:02.058584 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:02.059227 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:02.059228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:02.059228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:13:02.059232 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:02.059243 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:13:02.059247 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:02.059277 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:02.059381 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 05:13:02.059393 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:13:02.067123 22182 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:13:02.067257 22182 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:13:02.070745 22182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2979ad4ab2f843fd928dd7aaed3aeb92: Generated new TSK 0
I20260924 05:13:02.070922 22182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:13:02.073364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 22107
2026-09-24T05:13:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1260 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:13:02 dist-test-slave-mr2f krb5kdc[22206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:13:02 dist-test-slave-mr2f krb5kdc[22206](info): set up 2 sockets
Sep 24 05:13:02 dist-test-slave-mr2f krb5kdc[22206](info): commencing operation
krb5kdc: starting...
W20260924 05:13:03.090621 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:13:03 dist-test-slave-mr2f krb5kdc[22206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:13:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:13:03.199167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:13:03.276594 22219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:03.276788 22219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:03.276810 22219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:03.277889 22219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:03.277923 22219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:03.277937 22219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:03.277951 22219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:03.277962 22219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:03.279376 22219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:03.279572 22219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:03.279752 22219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:13:03.284348 22225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:03.284371 22227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:03.284433 22219 server_base.cc:1034] running on GCE node
W20260924 05:13:03.284390 22224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:03.284832 22219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:03.285070 22219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:03.286216 22219 hybrid_clock.cc:648] HybridClock initialized: now 1790226783286195 us; error 37 us; skew 500 ppm
Sep 24 05:13:03 dist-test-slave-mr2f krb5kdc[22206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:03.288540 22219 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:13:03.288992 22219 webserver.cc:460] Webserver started at http://127.0.98.62:40059/ using document root <none> and password file <none>
I20260924 05:13:03.289215 22219 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:03.289279 22219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:03.289423 22219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:03.290444 22219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "cf696b8ecc594295a5aa885dbb460265"
format_stamp: "Formatted at 2026-09-24 05:13:03 on dist-test-slave-mr2f"
server_key: "26c0931fed33ec51b45fa82bcb50ef8b"
server_key_iv: "a2f47cc2a68071c7dc78d9f74c221af8"
server_key_version: "encryptionkey@0"
I20260924 05:13:03.290843 22219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf696b8ecc594295a5aa885dbb460265"
format_stamp: "Formatted at 2026-09-24 05:13:03 on dist-test-slave-mr2f"
server_key: "26c0931fed33ec51b45fa82bcb50ef8b"
server_key_iv: "a2f47cc2a68071c7dc78d9f74c221af8"
server_key_version: "encryptionkey@0"
I20260924 05:13:03.292084 22219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:13:03.292794 22234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:03.292961 22219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:03.293057 22219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "cf696b8ecc594295a5aa885dbb460265"
format_stamp: "Formatted at 2026-09-24 05:13:03 on dist-test-slave-mr2f"
server_key: "26c0931fed33ec51b45fa82bcb50ef8b"
server_key_iv: "a2f47cc2a68071c7dc78d9f74c221af8"
server_key_version: "encryptionkey@0"
I20260924 05:13:03.293145 22219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:03.303165 22219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:03.305320 22219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:03.305470 22219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:03.309449 22219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44219
I20260924 05:13:03.309481 22286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44219 every 8 connection(s)
I20260924 05:13:03.309813 22219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:13:03.310606 22287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:03.312769 22287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265: Bootstrap starting.
I20260924 05:13:03.313333 22287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:03.313700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 22219
I20260924 05:13:03.313704 22287 log.cc:824] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:03.313805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:13:03.313995 392 external_mini_cluster.cc:1428] Setting key 0ceab935c719c67b9e758201e17ac5a1
I20260924 05:13:03.314482 22287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265: No bootstrap required, opened a new log
I20260924 05:13:03.316494 22287 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf696b8ecc594295a5aa885dbb460265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260924 05:13:03.316673 22287 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:13:03.316715 22287 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf696b8ecc594295a5aa885dbb460265, State: Initialized, Role: FOLLOWER
I20260924 05:13:03.316846 22287 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf696b8ecc594295a5aa885dbb460265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260924 05:13:03.316934 22287 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:13:03.316972 22287 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:13:03.317018 22287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:03.317677 22287 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf696b8ecc594295a5aa885dbb460265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260924 05:13:03.317781 22287 leader_election.cc:302] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf696b8ecc594295a5aa885dbb460265; no voters:
I20260924 05:13:03.317917 22287 leader_election.cc:288] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:13:03.317951 22292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:13:03.318102 22292 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [term 1 LEADER]: Becoming Leader. State: Replica: cf696b8ecc594295a5aa885dbb460265, State: Running, Role: LEADER
I20260924 05:13:03.318171 22287 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:03.318220 22292 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf696b8ecc594295a5aa885dbb460265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260924 05:13:03.318645 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf696b8ecc594295a5aa885dbb460265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf696b8ecc594295a5aa885dbb460265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } } }
I20260924 05:13:03.318722 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:03.318969 22294 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf696b8ecc594295a5aa885dbb460265. Latest consensus state: current_term: 1 leader_uuid: "cf696b8ecc594295a5aa885dbb460265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf696b8ecc594295a5aa885dbb460265" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } } }
I20260924 05:13:03.319216 22300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:13:03.319270 22294 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:03.319706 22300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:13:03.321372 22300 catalog_manager.cc:1347] Generated new cluster ID: a841e21ffeb4494d8dbd01dcfdffa702
I20260924 05:13:03.321424 22300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:13:03 dist-test-slave-mr2f krb5kdc[22206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:13:03.326042 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:03.314463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38443 (local address 127.0.98.62:44219)
0924 05:13:03.314669 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:03.314678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:03.314710 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:13:03.315725 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:03.315734 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:03.315795 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:03.315944 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:03.316540 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:03.318235 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:03.319462 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:03.320153 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:03.320258 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:03.322750 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:03.322753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:03.322754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:03.322790 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:03.324483 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:03.324864 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:03.324867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:03.324867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:03.324973 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:03.325198 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:03.325199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:03.325199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:03.325436 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:03.325604 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:03.325722 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 05:13:03.325859 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260924 05:13:03.326905 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:13:03.334079 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:03.328197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50939 (local address 127.0.98.62:44219)
0924 05:13:03.328232 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:03.328234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:03.328353 (+ 119us) server_negotiation.cc:407] Connection header received
0924 05:13:03.328423 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:03.328424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:03.328434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:03.328488 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:03.328975 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:03.329434 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:03.330120 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:03.330344 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:03.330430 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:03.331598 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:03.331605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:03.331606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:03.331612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:03.332972 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:03.333322 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:03.333323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:03.333323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:03.333375 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:03.333592 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:03.333593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:03.333593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:03.333704 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:03.333802 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:03.333901 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 05:13:03.333974 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:13:03.337532 22300 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:13:03.338232 22300 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:13:03.344004 22300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf696b8ecc594295a5aa885dbb460265: Generated new TSK 0
I20260924 05:13:03.344168 22300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:13:03.348342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 22219
2026-09-24T05:13:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:13:03 dist-test-slave-mr2f krb5kdc[22318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:13:03 dist-test-slave-mr2f krb5kdc[22318](info): set up 2 sockets
Sep 24 05:13:03 dist-test-slave-mr2f krb5kdc[22318](info): commencing operation
krb5kdc: starting...
W20260924 05:13:04.362726 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:13:04 dist-test-slave-mr2f krb5kdc[22318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:13:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:13:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:13:04.475883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:13:04.554087 22331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:04.554345 22331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:04.554383 22331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:04.555583 22331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:04.555660 22331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:04.555684 22331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:04.555712 22331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:04.555743 22331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:04.557220 22331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:04.557497 22331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:04.557721 22331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:13:04.562419 22336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:04.562413 22337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:04.562615 22331 server_base.cc:1034] running on GCE node
W20260924 05:13:04.562412 22339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:04.562898 22331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:04.563175 22331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:04.564338 22331 hybrid_clock.cc:648] HybridClock initialized: now 1790226784564322 us; error 28 us; skew 500 ppm
Sep 24 05:13:04 dist-test-slave-mr2f krb5kdc[22318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:04.566682 22331 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:13:04.567152 22331 webserver.cc:460] Webserver started at http://127.0.98.62:43521/ using document root <none> and password file <none>
I20260924 05:13:04.567374 22331 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:04.567442 22331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:04.567606 22331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:04.568652 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "01a33ec4637c4508ab4ace484a12f06e"
format_stamp: "Formatted at 2026-09-24 05:13:04 on dist-test-slave-mr2f"
server_key: "572b5db1604825575b8ea784e879727f"
server_key_iv: "0d160314e4899b2eb01500d6056df2db"
server_key_version: "encryptionkey@0"
I20260924 05:13:04.569000 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "01a33ec4637c4508ab4ace484a12f06e"
format_stamp: "Formatted at 2026-09-24 05:13:04 on dist-test-slave-mr2f"
server_key: "572b5db1604825575b8ea784e879727f"
server_key_iv: "0d160314e4899b2eb01500d6056df2db"
server_key_version: "encryptionkey@0"
I20260924 05:13:04.570298 22331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:13:04.571061 22346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:04.571216 22331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:13:04.571322 22331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "01a33ec4637c4508ab4ace484a12f06e"
format_stamp: "Formatted at 2026-09-24 05:13:04 on dist-test-slave-mr2f"
server_key: "572b5db1604825575b8ea784e879727f"
server_key_iv: "0d160314e4899b2eb01500d6056df2db"
server_key_version: "encryptionkey@0"
I20260924 05:13:04.571394 22331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:04.590320 22331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:04.592272 22331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:04.592435 22331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:04.596379 22331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40763
I20260924 05:13:04.596414 22398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40763 every 8 connection(s)
I20260924 05:13:04.596781 22331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:13:04.597580 22399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:04.599865 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e: Bootstrap starting.
I20260924 05:13:04.600417 22399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:04.600693 22399 log.cc:824] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:04.600777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 22331
I20260924 05:13:04.600947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:13:04.601135 392 external_mini_cluster.cc:1428] Setting key 7d01779b4a620f7d71a48daec2535855
I20260924 05:13:04.601488 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e: No bootstrap required, opened a new log
I20260924 05:13:04.603273 22399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a33ec4637c4508ab4ace484a12f06e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } }
I20260924 05:13:04.603440 22399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:13:04.603479 22399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a33ec4637c4508ab4ace484a12f06e, State: Initialized, Role: FOLLOWER
I20260924 05:13:04.603641 22399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a33ec4637c4508ab4ace484a12f06e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } }
I20260924 05:13:04.603739 22399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:13:04.603763 22399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:13:04.603833 22399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:04.604341 22399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a33ec4637c4508ab4ace484a12f06e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } }
I20260924 05:13:04.604424 22399 leader_election.cc:302] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01a33ec4637c4508ab4ace484a12f06e; no voters:
I20260924 05:13:04.604612 22399 leader_election.cc:288] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:13:04.604640 22404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:13:04.604751 22404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [term 1 LEADER]: Becoming Leader. State: Replica: 01a33ec4637c4508ab4ace484a12f06e, State: Running, Role: LEADER
I20260924 05:13:04.604872 22399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:04.604879 22404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a33ec4637c4508ab4ace484a12f06e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } }
I20260924 05:13:04.606520 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a33ec4637c4508ab4ace484a12f06e. Latest consensus state: current_term: 1 leader_uuid: "01a33ec4637c4508ab4ace484a12f06e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a33ec4637c4508ab4ace484a12f06e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } } }
W20260924 05:13:04.606642 22419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 05:13:04.606666 22405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01a33ec4637c4508ab4ace484a12f06e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a33ec4637c4508ab4ace484a12f06e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } } }
I20260924 05:13:04.606729 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [sys.catalog]: This master's current role is: LEADER
W20260924 05:13:04.606812 22419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:13:04.606907 22405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:04.607174 22421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:13:04.607354 22421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:13:04.608387 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:04.601603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33305 (local address 127.0.98.62:40763)
0924 05:13:04.601771 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:04.601779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:04.601803 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:13:04.603258 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:04.603264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:04.603311 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:04.603472 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:04.603710 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:04.605152 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:04.606202 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:04.606809 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:04.606905 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:04.607695 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:04.607699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:04.607699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:13:04.607902 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:04.607935 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:13:04.607949 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:04.608027 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:04.608102 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 05:13:04.608162 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260924 05:13:04.608954 22421 catalog_manager.cc:1347] Generated new cluster ID: 5bf283fbdf934fb2be5655d76ffaf362
I20260924 05:13:04.609107 22421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:13:04.609344 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:13:04.614022 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:04.610889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47519 (local address 127.0.98.62:40763)
0924 05:13:04.610922 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:04.610924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:04.610982 (+ 58us) server_negotiation.cc:407] Connection header received
0924 05:13:04.611054 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:04.611055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:04.611063 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:04.611087 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:04.611242 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:04.611896 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:04.612806 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:04.613074 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:04.613108 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:04.613806 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:04.613808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:04.613808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:13:04.613812 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:04.613824 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:13:04.613828 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:04.613851 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:04.613935 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 05:13:04.613955 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:13:04.616037 22421 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:13:04.616730 22421 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:13:04.627928 22421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01a33ec4637c4508ab4ace484a12f06e: Generated new TSK 0
I20260924 05:13:04.628118 22421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:13:04.637195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 22331
2026-09-24T05:13:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:13:04 dist-test-slave-mr2f krb5kdc[22430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:13:04 dist-test-slave-mr2f krb5kdc[22430](info): set up 2 sockets
Sep 24 05:13:04 dist-test-slave-mr2f krb5kdc[22430](info): commencing operation
krb5kdc: starting...
W20260924 05:13:05.642586 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:13:05 dist-test-slave-mr2f krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:13:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:13:05.753096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45991
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:13:05.831854 22443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:05.832082 22443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:05.832118 22443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:05.833261 22443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:05.833328 22443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:05.833352 22443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:05.833380 22443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:05.833412 22443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:05.834847 22443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:05.835129 22443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:05.835381 22443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:13:05.840063 22451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:05.840181 22443 server_base.cc:1034] running on GCE node
W20260924 05:13:05.840099 22449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:05.840287 22448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:05.840554 22443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:05.840840 22443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:05.842003 22443 hybrid_clock.cc:648] HybridClock initialized: now 1790226785841991 us; error 46 us; skew 500 ppm
Sep 24 05:13:05 dist-test-slave-mr2f krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:05.849610 22443 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:13:05.850075 22443 webserver.cc:460] Webserver started at http://127.0.98.62:36503/ using document root <none> and password file <none>
I20260924 05:13:05.850289 22443 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:05.850354 22443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:05.850498 22443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:05.851630 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "7b4da091ceaf478d9d7bb0c1a540ef21"
format_stamp: "Formatted at 2026-09-24 05:13:05 on dist-test-slave-mr2f"
server_key: "10b158340ffd31746b310132dfc17ae1"
server_key_iv: "d7bf37bbac85eec51d1e0ffe1cc36acc"
server_key_version: "encryptionkey@0"
I20260924 05:13:05.851980 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b4da091ceaf478d9d7bb0c1a540ef21"
format_stamp: "Formatted at 2026-09-24 05:13:05 on dist-test-slave-mr2f"
server_key: "10b158340ffd31746b310132dfc17ae1"
server_key_iv: "d7bf37bbac85eec51d1e0ffe1cc36acc"
server_key_version: "encryptionkey@0"
I20260924 05:13:05.853268 22443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:13:05.853981 22458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:05.854147 22443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:13:05.854245 22443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "7b4da091ceaf478d9d7bb0c1a540ef21"
format_stamp: "Formatted at 2026-09-24 05:13:05 on dist-test-slave-mr2f"
server_key: "10b158340ffd31746b310132dfc17ae1"
server_key_iv: "d7bf37bbac85eec51d1e0ffe1cc36acc"
server_key_version: "encryptionkey@0"
I20260924 05:13:05.854324 22443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:05.857622 22443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:05.859557 22443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:05.859712 22443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:05.863567 22443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45991
I20260924 05:13:05.863603 22510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45991 every 8 connection(s)
I20260924 05:13:05.863984 22443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:13:05.864929 22511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:05.867158 22511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21: Bootstrap starting.
I20260924 05:13:05.867759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 22443
I20260924 05:13:05.867796 22511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:05.867889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:13:05.868116 392 external_mini_cluster.cc:1428] Setting key 3a9b721e25d71b5e411b2b18f5eb50cb
I20260924 05:13:05.868124 22511 log.cc:824] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:05.868933 22511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21: No bootstrap required, opened a new log
I20260924 05:13:05.870499 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:05.868609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41937 (local address 127.0.98.62:45991)
0924 05:13:05.868775 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:05.868784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:05.868818 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:13:05.869269 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:05.869275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:05.869301 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:05.869444 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:05.870104 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:05.870107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:05.870107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:13:05.870111 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:05.870152 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:13:05.870163 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:05.870249 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:05.870306 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 05:13:05.870345 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260924 05:13:05.871407 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:13:05.871425 22511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4da091ceaf478d9d7bb0c1a540ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260924 05:13:05.871605 22511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:13:05.871652 22511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4da091ceaf478d9d7bb0c1a540ef21, State: Initialized, Role: FOLLOWER
I20260924 05:13:05.871766 22511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4da091ceaf478d9d7bb0c1a540ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260924 05:13:05.871877 22511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:13:05.871922 22511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:13:05.871968 22511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:05.872521 22511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4da091ceaf478d9d7bb0c1a540ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260924 05:13:05.872628 22511 leader_election.cc:302] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4da091ceaf478d9d7bb0c1a540ef21; no voters:
I20260924 05:13:05.872793 22511 leader_election.cc:288] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:13:05.872856 22520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:13:05.873035 22520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4da091ceaf478d9d7bb0c1a540ef21, State: Running, Role: LEADER
I20260924 05:13:05.873063 22511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:05.873162 22520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4da091ceaf478d9d7bb0c1a540ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } }
I20260924 05:13:05.873518 22520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b4da091ceaf478d9d7bb0c1a540ef21. Latest consensus state: current_term: 1 leader_uuid: "7b4da091ceaf478d9d7bb0c1a540ef21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4da091ceaf478d9d7bb0c1a540ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } } }
I20260924 05:13:05.873744 22520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:05.873523 22522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b4da091ceaf478d9d7bb0c1a540ef21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4da091ceaf478d9d7bb0c1a540ef21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45991 } } }
I20260924 05:13:05.874071 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:05.872805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34639 (local address 127.0.98.62:45991)
0924 05:13:05.872838 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:05.872839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:05.873020 (+ 181us) server_negotiation.cc:407] Connection header received
0924 05:13:05.873124 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:05.873125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:05.873129 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:05.873156 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:05.873797 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:05.873798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:05.873799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:13:05.873803 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:05.873816 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:13:05.873820 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:05.873850 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:05.873966 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 05:13:05.873979 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:13:05.873991 22522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21 [sys.catalog]: This master's current role is: LEADER
W20260924 05:13:05.875000 22537 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:13:05.875042 22537 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:13:05.875131 22530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:13:05.875304 22530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:13:05.876791 22530 catalog_manager.cc:1347] Generated new cluster ID: acb5ada8f31c425096eb8af5274d3764
I20260924 05:13:05.876847 22530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:13:05.891172 22530 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:13:05.891352 22530 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:13:05.896169 22530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b4da091ceaf478d9d7bb0c1a540ef21: Generated new TSK 0
I20260924 05:13:05.896342 22530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:13:05.904855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 22443
2026-09-24T05:13:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1268 ms)
[----------] 18 tests from AuthTokenIssuingTest (23041 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:13:05 dist-test-slave-mr2f krb5kdc[22542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:13:05 dist-test-slave-mr2f krb5kdc[22542](info): set up 2 sockets
Sep 24 05:13:05 dist-test-slave-mr2f krb5kdc[22542](info): commencing operation
krb5kdc: starting...
W20260924 05:13:06.946888 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:13:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:13:07.059070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46679,127.0.98.61:37369,127.0.98.60:35523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:13:07.136497 22555 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:13:07.136701 22555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:07.136719 22555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:07.136757 22555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:07.137785 22555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:07.137818 22555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:07.137832 22555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:07.137844 22555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:07.137856 22555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:07.139274 22555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46679,127.0.98.61:37369,127.0.98.60:35523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:07.139480 22555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:07.139636 22555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:13:07.144142 22561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:07.144148 22555 server_base.cc:1034] running on GCE node
W20260924 05:13:07.144248 22560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:07.144331 22563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:07.144586 22555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:07.144840 22555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:07.145987 22555 hybrid_clock.cc:648] HybridClock initialized: now 1790226787145970 us; error 35 us; skew 500 ppm
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:07.148363 22555 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:13:07.148819 22555 webserver.cc:460] Webserver started at http://127.0.98.62:32907/ using document root <none> and password file <none>
I20260924 05:13:07.149044 22555 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:07.149109 22555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:07.149253 22555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:07.150240 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "4159baa7642c48bab06bab090348c449"
format_stamp: "Formatted at 2026-09-24 05:13:07 on dist-test-slave-mr2f"
server_key: "9314ae9af3cbb470abb5edaf10987145"
server_key_iv: "3b4c5a2771444f2b668a609fbebb3407"
server_key_version: "encryptionkey@0"
I20260924 05:13:07.150579 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "4159baa7642c48bab06bab090348c449"
format_stamp: "Formatted at 2026-09-24 05:13:07 on dist-test-slave-mr2f"
server_key: "9314ae9af3cbb470abb5edaf10987145"
server_key_iv: "3b4c5a2771444f2b668a609fbebb3407"
server_key_version: "encryptionkey@0"
I20260924 05:13:07.151849 22555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:13:07.152506 22570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:07.152678 22555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:13:07.152769 22555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "4159baa7642c48bab06bab090348c449"
format_stamp: "Formatted at 2026-09-24 05:13:07 on dist-test-slave-mr2f"
server_key: "9314ae9af3cbb470abb5edaf10987145"
server_key_iv: "3b4c5a2771444f2b668a609fbebb3407"
server_key_version: "encryptionkey@0"
I20260924 05:13:07.152853 22555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:07.171815 22555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:07.173437 22555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:07.173596 22555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:07.177587 22555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46679
I20260924 05:13:07.177623 22622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46679 every 8 connection(s)
I20260924 05:13:07.177946 22555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:13:07.178860 22623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:07.180292 22623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:07.183565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 22555
I20260924 05:13:07.183666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:13:07.183884 392 external_mini_cluster.cc:1428] Setting key b93e84b0d9e19e5a819fc7853ab25b6f
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:13:07.189905 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.180959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.0.98.62:46679)
0924 05:13:07.181183 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.181186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.181210 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:13:07.181658 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.181659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.181675 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.181742 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.182185 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.183016 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.184132 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.184421 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.184506 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.186723 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.186729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.186730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.186736 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.188344 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.188836 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.188838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.188838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.188910 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.189167 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.189168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.189168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.189355 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.189415 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.189660 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.189756 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":66,"threads_started":1}
I20260924 05:13:07.190009 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.180889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46679 (local address 127.0.0.1:50526)
0924 05:13:07.181082 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.181112 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:07.181517 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:07.181747 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:07.181749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:07.181968 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:07.182108 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.182112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.183093 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.183094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.184053 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.184056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.184525 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.184526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.184624 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.185079 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:07.185089 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:07.186634 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:07.188440 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.188448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.188450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.188751 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.188979 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.188980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.188980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.189096 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.189427 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:07.189435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:07.189511 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 05:13:07.189634 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 05:13:07.189848 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":60,"thread_start_us":85,"threads_started":1}
I20260924 05:13:07.190800 22623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:07.191154 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.191035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37369 (local address 127.0.0.1:45778)
0924 05:13:07.191085 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.191136 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 05:13:07.191320 22574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
I20260924 05:13:07.191708 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.191626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37369 (local address 127.0.0.1:45794)
0924 05:13:07.191671 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.191693 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
W20260924 05:13:07.191826 22623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37369: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 05:13:07.208845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37369
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46679,127.0.98.61:37369,127.0.98.60:35523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:13:07.241117 22623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } attempt: 1
I20260924 05:13:07.241626 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.241490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37369 (local address 127.0.0.1:45804)
0924 05:13:07.241556 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.241599 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 05:13:07.241977 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.241893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37369 (local address 127.0.0.1:45814)
0924 05:13:07.241934 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.241962 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 05:13:07.242082 22623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37369: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
W20260924 05:13:07.286531 22630 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:13:07.286729 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:07.286746 22630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:07.286825 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:07.287933 22630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:07.287971 22630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:07.287985 22630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:07.287997 22630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:07.288035 22630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:07.289424 22630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46679,127.0.98.61:37369,127.0.98.60:35523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:07.289649 22630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:07.289841 22630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:13:07.294293 22636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:07.294451 22630 server_base.cc:1034] running on GCE node
W20260924 05:13:07.294319 22638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:07.294286 22635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:07.294795 22630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:07.295130 22630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:07.296273 22630 hybrid_clock.cc:648] HybridClock initialized: now 1790226787296260 us; error 35 us; skew 500 ppm
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:07.298661 22630 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 05:13:07.299129 22630 webserver.cc:460] Webserver started at http://127.0.98.61:37897/ using document root <none> and password file <none>
I20260924 05:13:07.299340 22630 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:07.299382 22630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:07.299535 22630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:07.300529 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/data/instance:
uuid: "d07641a137c04f838fb439fe4cf42e57"
format_stamp: "Formatted at 2026-09-24 05:13:07 on dist-test-slave-mr2f"
server_key: "353307bc25e7a36965b9217f5c859a12"
server_key_iv: "9b25e0413a56506873eaf06499976a5b"
server_key_version: "encryptionkey@0"
I20260924 05:13:07.300873 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/wal/instance:
uuid: "d07641a137c04f838fb439fe4cf42e57"
format_stamp: "Formatted at 2026-09-24 05:13:07 on dist-test-slave-mr2f"
server_key: "353307bc25e7a36965b9217f5c859a12"
server_key_iv: "9b25e0413a56506873eaf06499976a5b"
server_key_version: "encryptionkey@0"
I20260924 05:13:07.302103 22630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:13:07.302817 22645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:07.302958 22630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:07.303051 22630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/wal
uuid: "d07641a137c04f838fb439fe4cf42e57"
format_stamp: "Formatted at 2026-09-24 05:13:07 on dist-test-slave-mr2f"
server_key: "353307bc25e7a36965b9217f5c859a12"
server_key_iv: "9b25e0413a56506873eaf06499976a5b"
server_key_version: "encryptionkey@0"
I20260924 05:13:07.303102 22630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:07.310216 22623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } attempt: 2
I20260924 05:13:07.310657 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.310540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37369 (local address 127.0.0.1:45826)
0924 05:13:07.310594 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.310630 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:13:07.310940 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.310867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37369 (local address 127.0.0.1:45840)
0924 05:13:07.310890 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.310923 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260924 05:13:07.311048 22623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37369: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37369: connect: Connection refused (error 111)
I20260924 05:13:07.315930 22630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:07.318115 22630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:07.318265 22630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:07.321924 22630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37369
I20260924 05:13:07.321991 22697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37369 every 8 connection(s)
I20260924 05:13:07.322314 22630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:13:07.322979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 22630
I20260924 05:13:07.323107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-1/wal/instance
I20260924 05:13:07.323341 392 external_mini_cluster.cc:1428] Setting key 1f192d960fcd89434f930b5576afb038
I20260924 05:13:07.323540 22698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:07.325577 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:13:07.337288 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.326262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.0.98.62:46679)
0924 05:13:07.326330 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.326332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.326573 (+ 241us) server_negotiation.cc:407] Connection header received
0924 05:13:07.327295 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.327296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.327306 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.327351 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.328192 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.328651 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.330092 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.330271 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.330314 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.334501 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.334503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.334503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.334510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.335963 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.336333 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.336335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.336335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.336398 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.336660 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.336661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.336662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.336775 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.336846 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.337113 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.337197 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:13:07.337471 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.326242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46679 (local address 127.0.0.1:50532)
0924 05:13:07.326433 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.326480 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:07.327191 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:07.327423 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:07.327427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:07.327793 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:07.328107 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.328114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.328736 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.328737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.330005 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.330008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.330361 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.330363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.330513 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.331009 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:07.331018 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:07.334396 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:07.336037 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.336042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.336044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.336247 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.336471 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.336472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.336472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.336585 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.336849 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:07.336856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:07.336984 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 05:13:07.337092 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 05:13:07.337241 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260924 05:13:07.338058 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:13:07.345949 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.338336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37369 (local address 127.0.0.1:45856)
0924 05:13:07.338390 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.338399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:07.338438 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:07.338693 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:07.338694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:07.338856 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:07.338896 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.338897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.339733 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.339734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.340246 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.340247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.340787 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.340789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.340864 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.341179 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:07.341183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:07.343154 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:07.344911 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.344912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.344912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.345080 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.345295 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.345296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.345297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.345351 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.345662 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:07.345663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:07.345699 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 05:13:07.345757 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:13:07.345848 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 05:13:07.346010 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.338411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.0.98.61:37369)
0924 05:13:07.338566 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.338570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.338584 (+ 14us) server_negotiation.cc:407] Connection header received
0924 05:13:07.338600 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.338602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.338611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.338693 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.338967 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.339639 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.340331 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.340696 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.340767 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.343224 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.343231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.343231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.343237 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.344845 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.345172 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.345174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.345174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.345221 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.345419 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.345420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.345421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.345546 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.345683 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.345793 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.345895 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260924 05:13:07.346585 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:07.346944 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.346841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49394)
0924 05:13:07.346885 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.346924 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:13:07.347081 22648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 05:13:07.347474 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.347400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49410)
0924 05:13:07.347435 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.347458 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 05:13:07.347556 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35523: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
I20260924 05:13:07.347874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35523
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46679,127.0.98.61:37369,127.0.98.60:35523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:13:07.396872 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } attempt: 1
I20260924 05:13:07.397359 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.397217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49420)
0924 05:13:07.397292 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.397331 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 05:13:07.397688 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.397599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49426)
0924 05:13:07.397641 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.397663 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:13:07.397773 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35523: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
I20260924 05:13:07.402201 22623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } attempt: 3
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:13:07.409124 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.402461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37369 (local address 127.0.0.1:45868)
0924 05:13:07.402508 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.402522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:07.402592 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:07.402735 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:07.402735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:07.402932 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:07.402993 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.402995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.403697 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.403698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.404272 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.404273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.404671 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.404672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.404750 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.405221 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:07.405225 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:07.406395 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:07.408116 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.408117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.408117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.408289 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.408489 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.408490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.408490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.408548 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.408801 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:07.408802 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:07.408848 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:13:07.408933 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 05:13:07.409029 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:13:07.409124 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.402523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.0.98.61:37369)
0924 05:13:07.402590 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.402592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.402603 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:13:07.402653 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.402654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.402665 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.402730 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.403095 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.403587 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.404363 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.404584 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.404671 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.406483 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.406485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.406485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.406493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.407982 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.408362 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.408363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.408363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.408422 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.408622 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.408623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.408624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.408733 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.408809 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.408957 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.409029 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 05:13:07.409749 22623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:07.410092 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.409988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49432)
0924 05:13:07.410033 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.410068 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:13:07.410379 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.410299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49440)
0924 05:13:07.410338 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.410360 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:13:07.410471 22623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35523: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
W20260924 05:13:07.427739 22705 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:13:07.427919 22705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:07.427937 22705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:07.428020 22705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:07.429244 22705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:07.429283 22705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:07.429297 22705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:07.429348 22705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:07.429370 22705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:07.430737 22705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46679,127.0.98.61:37369,127.0.98.60:35523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:07.431051 22705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:07.431273 22705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:13:07.435742 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:07.435743 22710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:07.435746 22711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:07.435863 22705 server_base.cc:1034] running on GCE node
I20260924 05:13:07.436121 22705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:07.436398 22705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:07.437539 22705 hybrid_clock.cc:648] HybridClock initialized: now 1790226787437519 us; error 36 us; skew 500 ppm
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:07.439841 22705 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 05:13:07.440263 22705 webserver.cc:460] Webserver started at http://127.0.98.60:43869/ using document root <none> and password file <none>
I20260924 05:13:07.440474 22705 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:07.440539 22705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:07.440682 22705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:07.441592 22623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } attempt: 1
I20260924 05:13:07.441656 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/data/instance:
uuid: "91b124c079d64b13ba761685f8105554"
format_stamp: "Formatted at 2026-09-24 05:13:07 on dist-test-slave-mr2f"
server_key: "c860ef0603ec7fe6b339a55a0ae55656"
server_key_iv: "fb900d0e1444f787aace442d69281d42"
server_key_version: "encryptionkey@0"
I20260924 05:13:07.441929 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.441801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49448)
0924 05:13:07.441851 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.441884 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:13:07.442094 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/wal/instance:
uuid: "91b124c079d64b13ba761685f8105554"
format_stamp: "Formatted at 2026-09-24 05:13:07 on dist-test-slave-mr2f"
server_key: "c860ef0603ec7fe6b339a55a0ae55656"
server_key_iv: "fb900d0e1444f787aace442d69281d42"
server_key_version: "encryptionkey@0"
I20260924 05:13:07.442219 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.442129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49460)
0924 05:13:07.442174 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.442196 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 05:13:07.442301 22623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35523: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35523: connect: Connection refused (error 111)
I20260924 05:13:07.443461 22705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:13:07.444195 22720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:07.444357 22705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:13:07.444453 22705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/wal
uuid: "91b124c079d64b13ba761685f8105554"
format_stamp: "Formatted at 2026-09-24 05:13:07 on dist-test-slave-mr2f"
server_key: "c860ef0603ec7fe6b339a55a0ae55656"
server_key_iv: "fb900d0e1444f787aace442d69281d42"
server_key_version: "encryptionkey@0"
I20260924 05:13:07.444532 22705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:07.458269 22705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:07.459955 22705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:07.460107 22705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:07.464049 22705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35523
I20260924 05:13:07.464100 22772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35523 every 8 connection(s)
I20260924 05:13:07.464459 22705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:13:07.465370 22773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:07.465906 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } attempt: 2
I20260924 05:13:07.467288 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:13:07.471830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 22705
I20260924 05:13:07.471930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790226527740525-392-0/minicluster-data/master-2/wal/instance
I20260924 05:13:07.472110 392 external_mini_cluster.cc:1428] Setting key e24ac52c29c655cc99138f7020cf7c7c
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:13:07.475875 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.466170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49470)
0924 05:13:07.466250 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.466269 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:07.466318 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:07.467376 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:07.467378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:07.467571 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:07.467621 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.467623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.469051 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.469052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.469656 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.469658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.470346 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.470347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.470410 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.470872 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:07.470877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:07.472174 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:07.474488 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.474489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.474489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.474709 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.475030 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.475031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.475031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.475094 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.475557 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:07.475558 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:07.475603 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 05:13:07.475651 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 05:13:07.475759 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 05:13:07.476282 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.467940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.0.98.62:46679)
0924 05:13:07.467987 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.467989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.468072 (+ 83us) server_negotiation.cc:407] Connection header received
0924 05:13:07.468142 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.468142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.468150 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.468195 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.468740 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.469289 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.470194 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.470414 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.470484 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.473053 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.473054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.473054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.473079 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.474904 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.475348 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.475349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.475350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.475430 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.475644 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.475645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.475645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.475754 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.475836 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.476111 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.476194 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:13:07.476517 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.467828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46679 (local address 127.0.0.1:50534)
0924 05:13:07.467976 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.468011 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:07.468083 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:07.468213 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:07.468216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:07.468535 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:07.468650 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.468655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.469375 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.469376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.470116 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.470117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.470508 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.470509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.470599 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.471069 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:07.471076 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:07.472921 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:07.474993 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.474999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.475001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.475258 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.475507 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.475508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.475508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.475578 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.475866 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:07.475872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:07.475942 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 05:13:07.476065 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 05:13:07.476227 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260924 05:13:07.476519 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.466383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.0.98.60:35523)
0924 05:13:07.466565 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.466574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.466605 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:13:07.467156 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.467165 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.467233 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.467401 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.467748 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.468963 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.469732 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.470273 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.470344 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.472264 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.472271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.472272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.472280 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.474372 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.474829 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.474832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.474832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.474933 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.475164 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.475164 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.475165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.475471 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.476095 (+ 624us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.476144 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.476312 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260924 05:13:07.477227 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:13:07.478499 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57: Bootstrap starting.
I20260924 05:13:07.480010 22698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:07.480332 22698 log.cc:824] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:07.481168 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57: No bootstrap required, opened a new log
I20260924 05:13:07.481604 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.472437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.0.98.60:35523)
0924 05:13:07.472592 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.472593 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.472758 (+ 165us) server_negotiation.cc:407] Connection header received
0924 05:13:07.472847 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.472848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.472857 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.472925 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.473686 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.474304 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.475135 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.475439 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.475508 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.478743 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.478745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.478745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.478751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.480142 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.480514 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.480515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.480516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.480574 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.481019 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.481020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.481021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.481148 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.481294 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.481375 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.481447 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:13:07.483464 22698 raft_consensus.cc:348] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } }
I20260924 05:13:07.483606 22698 raft_consensus.cc:729] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d07641a137c04f838fb439fe4cf42e57, State: Initialized, Role: FOLLOWER
I20260924 05:13:07.483686 22698 consensus_queue.cc:260] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } }
I20260924 05:13:07.483958 22780 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.484016 22698 sys_catalog.cc:564] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:07.484041 22780 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:07.485803 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.477532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.0.98.61:37369)
0924 05:13:07.477587 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.477590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.477598 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:13:07.477683 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.477684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.477696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.477762 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.478054 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.478705 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.480071 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.480333 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.480406 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.482526 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.482528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.482528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.482535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.484085 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.484786 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.484787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.484787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.484847 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.485099 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.485100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.485100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.485271 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.485318 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.485626 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.485701 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 05:13:07.485940 22791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:13:07.485993 22791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:13:07.486809 22630 master_runner.cc:186] Error getting master registration for 127.0.98.62:46679: OK, instance_id { permanent_uuid: "4159baa7642c48bab06bab090348c449" instance_seqno: 1790226787176489 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:13:07.488749 22630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:13:07.489243 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.477467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37369 (local address 127.0.0.1:45878)
0924 05:13:07.477506 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.477515 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:07.477615 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:07.477776 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:07.477776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:07.477908 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:07.477961 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.477963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.478825 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.478826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.479977 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.479980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.480484 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.480485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.480558 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.480904 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:07.480908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:07.482424 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:07.484438 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.484439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.484440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.484676 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.484924 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.484925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.484925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.485007 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.485374 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:07.485375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:07.485429 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 05:13:07.489122 (+ 3693us) client_negotiation.cc:240] Negotiation successful
0924 05:13:07.489189 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 05:13:07.489928 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:07.492443 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.482143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.0.98.62:46679)
0924 05:13:07.482176 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.482178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.482185 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:13:07.482246 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.482246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.482254 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.482303 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.482960 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.483539 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.484366 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.484654 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.484728 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.489137 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.489138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.489138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.489145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.491140 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.491556 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.491558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.491558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.491619 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.491866 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.491867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.491867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.491975 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.492138 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.492255 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.492327 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:13:07.498164 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.490171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49496)
0924 05:13:07.490233 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.490243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:07.490283 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:07.490405 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:07.490406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:07.490562 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:07.490594 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.490596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.491652 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.491653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.492465 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.492485 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.493161 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.493161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.493211 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.493627 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:07.493633 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:07.494979 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:07.496937 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.496938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.496939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.497157 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.497370 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.497370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.497371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.497433 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.497804 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:07.497805 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:07.497853 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 05:13:07.497943 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 05:13:07.498036 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:13:07.498216 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.490203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.0.98.60:35523)
0924 05:13:07.490238 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.490240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.490272 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:13:07.490328 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.490329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.490341 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.490400 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.490693 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.491566 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.492692 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.492985 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.493025 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.495105 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.495106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.495107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.495114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.496854 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.497236 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.497237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.497238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.497296 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.497522 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.497523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.497523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.497717 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.497813 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.497967 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.498086 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":16}
I20260924 05:13:07.499944 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554: Bootstrap starting.
I20260924 05:13:07.500662 22773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:07.500973 22773 log.cc:824] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:07.501681 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554: No bootstrap required, opened a new log
I20260924 05:13:07.501864 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.493695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.0.98.61:37369)
0924 05:13:07.493728 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.493730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.493737 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:13:07.493756 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.493757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.493784 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.493879 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.494327 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.494901 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.495847 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.496114 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.496175 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.498973 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.498974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.498975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.498982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.500574 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.500948 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.500949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.500949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.501005 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.501280 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.501281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.501282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.501401 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.501505 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.501712 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.501788 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:13:07.503474 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:13:07.503909 22773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } }
I20260924 05:13:07.504065 22773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b124c079d64b13ba761685f8105554, State: Initialized, Role: FOLLOWER
I20260924 05:13:07.504135 22773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } }
I20260924 05:13:07.504426 22773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:07.504390 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.504487 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:07.504771 22597 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4159baa7642c48bab06bab090348c449"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50546
W20260924 05:13:07.506629 22805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:13:07.506865 22805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:13:07.506923 22705 master_runner.cc:186] Error getting master registration for 127.0.98.62:46679: OK, instance_id { permanent_uuid: "4159baa7642c48bab06bab090348c449" instance_seqno: 1790226787176489 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:13:07.509043 22705 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:13:07.517458 22623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } attempt: 2
Sep 24 05:13:07 dist-test-slave-mr2f krb5kdc[22542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:13:07.525018 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.517688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35523 (local address 127.0.0.1:49504)
0924 05:13:07.517737 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:07.517750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:07.517799 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:07.517948 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:07.517949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:07.518111 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:07.518153 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.518155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.519060 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.519061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.519852 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:07.519854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.520326 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.520327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:07.520389 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.520805 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:07.520810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:07.522163 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:07.523960 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.523962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.523962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.524189 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.524390 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:07.524391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:07.524391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:07.524456 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:07.524709 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:07.524710 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:07.524758 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 05:13:07.524828 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:13:07.524918 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 05:13:07.525025 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.517791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.0.98.60:35523)
0924 05:13:07.517830 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.517832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.517840 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:13:07.517870 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.517871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.517879 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.517937 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.518229 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.518967 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.519941 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.520231 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.520282 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.522249 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.522250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.522250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.522257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.523875 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.524259 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.524259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.524260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.524317 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.524523 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.524524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.524524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.524644 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.524706 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.524862 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.524938 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:13:07.526980 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449: Bootstrap starting.
I20260924 05:13:07.527674 22623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:07.527963 22623 log.cc:824] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:07.528919 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449: No bootstrap required, opened a new log
I20260924 05:13:07.530253 22623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } }
I20260924 05:13:07.530385 22623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4159baa7642c48bab06bab090348c449, State: Initialized, Role: FOLLOWER
I20260924 05:13:07.530459 22623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } }
I20260924 05:13:07.530799 22623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:07.530859 22597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 05:13:07.530913 22597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:07.531652 22597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } }
I20260924 05:13:07.531880 22597 leader_election.cc:288] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [CANDIDATE]: Term 1 election: Requested vote from peers d07641a137c04f838fb439fe4cf42e57 (127.0.98.61:37369), 91b124c079d64b13ba761685f8105554 (127.0.98.60:35523)
I20260924 05:13:07.532261 22672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4159baa7642c48bab06bab090348c449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d07641a137c04f838fb439fe4cf42e57"
I20260924 05:13:07.532418 22672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:07.533326 22672 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4159baa7642c48bab06bab090348c449 in term 1.
I20260924 05:13:07.533301 22747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4159baa7642c48bab06bab090348c449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "91b124c079d64b13ba761685f8105554"
I20260924 05:13:07.533443 22747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:07.533650 22574 leader_election.cc:302] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4159baa7642c48bab06bab090348c449, d07641a137c04f838fb439fe4cf42e57; no voters:
I20260924 05:13:07.533910 22824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:13:07.534041 22824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [term 1 LEADER]: Becoming Leader. State: Replica: 4159baa7642c48bab06bab090348c449, State: Running, Role: LEADER
I20260924 05:13:07.534135 22824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } }
I20260924 05:13:07.534251 22747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4159baa7642c48bab06bab090348c449 in term 1.
I20260924 05:13:07.534521 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4159baa7642c48bab06bab090348c449. Latest consensus state: current_term: 1 leader_uuid: "4159baa7642c48bab06bab090348c449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.530792 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.534964 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:07.535041 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:07.535315 22555 master_runner.cc:263] No leader master found from master 4159baa7642c48bab06bab090348c449
I20260924 05:13:07.535406 22555 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 05:13:07.535880 22819 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:13:07.535985 22819 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:13:07.539548 22827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:13:07.539742 22827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:13:07.547113 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.539051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.98.61:37369)
0924 05:13:07.539092 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.539094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.539471 (+ 377us) server_negotiation.cc:407] Connection header received
0924 05:13:07.539576 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.539577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.539599 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.539628 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.540488 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.541211 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.542235 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.542530 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.542570 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.544184 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.544185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.544185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.544192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.545825 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.546214 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.546215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.546216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.546264 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.546519 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.546520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.546520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.546642 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.546709 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.546929 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.547018 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:13:07.557572 22747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [term 1 FOLLOWER]: Refusing update from remote peer 4159baa7642c48bab06bab090348c449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:13:07.557946 22825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:13:07.559303 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4159baa7642c48bab06bab090348c449. Latest consensus state: current_term: 1 leader_uuid: "4159baa7642c48bab06bab090348c449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.559399 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:07.559826 22672 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [term 1 FOLLOWER]: Refusing update from remote peer 4159baa7642c48bab06bab090348c449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:13:07.560075 22825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:13:07.561184 22780 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4159baa7642c48bab06bab090348c449. Latest consensus state: current_term: 1 leader_uuid: "4159baa7642c48bab06bab090348c449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.561271 22780 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:07.565003 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.539125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.0.98.62:46679)
0924 05:13:07.539170 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.539172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.548997 (+ 9825us) server_negotiation.cc:407] Connection header received
0924 05:13:07.549089 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.549090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.549100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.549127 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.550853 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.551544 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.554405 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.554702 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.554741 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.559771 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.559772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.559773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.559779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.561470 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.563922 (+ 2452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.563923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.563924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.563979 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.564164 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.564164 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.564165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.564289 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.564380 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.564822 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.564901 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:13:07.566041 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:07.539247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.0.98.60:35523)
0924 05:13:07.539281 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.539282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.548158 (+ 8876us) server_negotiation.cc:407] Connection header received
0924 05:13:07.548263 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.548264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.548274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.548302 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.549743 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.550428 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.553063 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:07.553349 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:07.553398 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:07.561918 (+ 8520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:07.561919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:07.561920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:07.561927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:07.563530 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.564560 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.564561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.564562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.564617 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:07.565497 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:07.565498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:07.565498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:07.565627 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:07.565711 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:07.565891 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 05:13:07.565960 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:13:07.568063 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4159baa7642c48bab06bab090348c449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.568148 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:07.568253 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4159baa7642c48bab06bab090348c449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.568320 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:07.568524 22832 mvcc.cc:205] Tried to move back new op lower bound from 7332768921821503488 to 7332768921740537856. Current Snapshot: MvccSnapshot[applied={T|T < 7332768921821503488}]
I20260924 05:13:07.568730 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4159baa7642c48bab06bab090348c449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.568807 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449 [sys.catalog]: This master's current role is: LEADER
W20260924 05:13:07.568832 22651 tablet.cc:2369] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:13:07.569132 22780 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4159baa7642c48bab06bab090348c449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4159baa7642c48bab06bab090348c449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46679 } } peers { permanent_uuid: "d07641a137c04f838fb439fe4cf42e57" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37369 } } peers { permanent_uuid: "91b124c079d64b13ba761685f8105554" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35523 } } }
I20260924 05:13:07.569209 22780 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:07.569273 22835 mvcc.cc:205] Tried to move back new op lower bound from 7332768921821503488 to 7332768921740537856. Current Snapshot: MvccSnapshot[applied={T|T < 7332768921821503488}]
I20260924 05:13:07.571581 22827 catalog_manager.cc:1347] Generated new cluster ID: 9c457e3bc4db4d9f9e3a28714b29f5b0
I20260924 05:13:07.571640 22827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:13:07.586974 22827 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:13:07.588183 22827 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:13:07.592794 22827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4159baa7642c48bab06bab090348c449: Generated new TSK 0
I20260924 05:13:07.592978 22827 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:13:07.614267 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.612547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.0.98.61:37369)
0924 05:13:07.612597 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.612599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.613557 (+ 958us) server_negotiation.cc:407] Connection header received
0924 05:13:07.613674 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.613675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.613687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.613718 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.614193 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45906: BlockingRecv error: recv got EOF from 127.0.0.1:45906 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 05:13:07.615034 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.612926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.0.98.60:35523)
0924 05:13:07.612967 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.612969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.614500 (+ 1531us) server_negotiation.cc:407] Connection header received
0924 05:13:07.614599 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.614600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.614610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.614640 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.614979 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49524: BlockingRecv error: recv got EOF from 127.0.0.1:49524 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 05:13:07.615911 22626 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.613280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.0.98.62:46679)
0924 05:13:07.613320 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.613321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.615246 (+ 1925us) server_negotiation.cc:407] Connection header received
0924 05:13:07.615456 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.615457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.615469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.615500 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.615846 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50572: BlockingRecv error: recv got EOF from 127.0.0.1:50572 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:13:07.618204 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.617428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.0.98.61:37369)
0924 05:13:07.617460 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.617462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.617698 (+ 236us) server_negotiation.cc:407] Connection header received
0924 05:13:07.617800 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.617801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.617817 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.617846 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.618162 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45908: BlockingRecv error: recv got EOF from 127.0.0.1:45908 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:13:07.621091 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.620310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.0.98.61:37369)
0924 05:13:07.620340 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.620342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.620604 (+ 262us) server_negotiation.cc:407] Connection header received
0924 05:13:07.620699 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.620700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.620709 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.620737 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.621034 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45920: BlockingRecv error: recv got EOF from 127.0.0.1:45920 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 05:13:07.624931 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.624154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.0.98.61:37369)
0924 05:13:07.624191 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.624193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.624445 (+ 252us) server_negotiation.cc:407] Connection header received
0924 05:13:07.624539 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.624540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.624550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.624581 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.624882 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45932: BlockingRecv error: recv got EOF from 127.0.0.1:45932 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:13:07.630810 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.629940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.0.98.61:37369)
0924 05:13:07.629973 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.629975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.630273 (+ 298us) server_negotiation.cc:407] Connection header received
0924 05:13:07.630369 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.630370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.630379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.630407 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.630734 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45942: BlockingRecv error: recv got EOF from 127.0.0.1:45942 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 05:13:07.640758 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.639889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.0.98.61:37369)
0924 05:13:07.639926 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.639928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.640169 (+ 241us) server_negotiation.cc:407] Connection header received
0924 05:13:07.640268 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.640269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.640279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.640307 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.640709 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45946: BlockingRecv error: recv got EOF from 127.0.0.1:45946 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 05:13:07.658885 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.657948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.0.98.61:37369)
0924 05:13:07.657990 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.657992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.658262 (+ 270us) server_negotiation.cc:407] Connection header received
0924 05:13:07.658359 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.658360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.658371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.658413 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.658764 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45948: BlockingRecv error: recv got EOF from 127.0.0.1:45948 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:13:07.692896 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.692169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.0.98.61:37369)
0924 05:13:07.692223 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.692225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.692354 (+ 129us) server_negotiation.cc:407] Connection header received
0924 05:13:07.692430 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.692431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.692442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.692506 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.692837 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45954: BlockingRecv error: recv got EOF from 127.0.0.1:45954 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 05:13:07.758625 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.757948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.0.98.61:37369)
0924 05:13:07.758031 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.758032 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.758118 (+ 86us) server_negotiation.cc:407] Connection header received
0924 05:13:07.758176 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.758176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.758187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.758230 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.758545 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45958: BlockingRecv error: recv got EOF from 127.0.0.1:45958 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 05:13:07.888352 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:07.887730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.0.98.61:37369)
0924 05:13:07.887811 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:07.887813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:07.887868 (+ 55us) server_negotiation.cc:407] Connection header received
0924 05:13:07.887925 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:07.887926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:07.887936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:07.887977 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:07.888280 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45960: BlockingRecv error: recv got EOF from 127.0.0.1:45960 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 05:13:08.146229 22703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:08.145675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.0.98.61:37369)
0924 05:13:08.145746 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:08.145748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:08.145758 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:13:08.145815 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:08.145816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:08.145826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:08.145866 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:08.146141 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45962: BlockingRecv error: recv got EOF from 127.0.0.1:45962 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 05:13:08.486398 22791 catalog_manager.cc:1259] Loaded cluster ID: 9c457e3bc4db4d9f9e3a28714b29f5b0
I20260924 05:13:08.486469 22791 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57: loading cluster ID for follower catalog manager: success
I20260924 05:13:08.487207 22791 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57: acquiring CA information for follower catalog manager: success
I20260924 05:13:08.487437 22791 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d07641a137c04f838fb439fe4cf42e57: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:13:08.507290 22805 catalog_manager.cc:1259] Loaded cluster ID: 9c457e3bc4db4d9f9e3a28714b29f5b0
I20260924 05:13:08.507355 22805 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554: loading cluster ID for follower catalog manager: success
I20260924 05:13:08.508134 22805 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554: acquiring CA information for follower catalog manager: success
I20260924 05:13:08.508423 22805 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91b124c079d64b13ba761685f8105554: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:13:08.661985 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:08.659438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.0.98.61:37369)
0924 05:13:08.659565 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:08.659567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:08.659579 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:13:08.659641 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:08.659642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:08.659699 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:08.659774 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:08.659925 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.660406 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.661179 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.661374 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.661456 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:08.661633 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:08.661745 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:08.661823 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:08.661862 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 05:13:08.661884 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":49,"threads_started":1}
I20260924 05:13:08.665980 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:08.663877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.0.98.60:35523)
0924 05:13:08.663994 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:08.663996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:08.664007 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:13:08.664049 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:08.664050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:08.664082 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:08.664148 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:08.664230 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.664686 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.665372 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.665556 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.665606 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:08.665723 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:08.665811 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:08.665864 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:08.665899 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 05:13:08.665915 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":72,"threads_started":1}
I20260924 05:13:08.670519 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:08.667669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.0.98.60:35523)
0924 05:13:08.667712 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:08.667714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:08.667848 (+ 134us) server_negotiation.cc:407] Connection header received
0924 05:13:08.667934 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:08.667935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:08.667944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:08.668004 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:08.668111 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.668817 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.669829 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.670033 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.670064 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:08.670255 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:08.670285 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:08.670333 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:08.670423 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 05:13:08.670446 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:13:08.670610 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:08.667562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.0.98.61:37369)
0924 05:13:08.667716 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:08.667719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:08.667849 (+ 130us) server_negotiation.cc:407] Connection header received
0924 05:13:08.668058 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:08.668060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:08.668069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:08.668116 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:08.668267 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.668710 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.669776 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.670078 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.670141 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:08.670317 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:08.670353 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:08.670415 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:08.670512 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 05:13:08.670537 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260924 05:13:08.671015 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:08.667615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.0.98.62:46679)
0924 05:13:08.667744 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:08.667746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:08.667822 (+ 76us) server_negotiation.cc:407] Connection header received
0924 05:13:08.667906 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:08.667907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:08.667958 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:08.668040 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:08.668222 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.668956 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.669897 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:08.670210 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:08.670260 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:08.670515 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:08.670622 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:08.670682 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:08.670843 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 05:13:08.670866 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260924 05:13:08.672564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 22555
I20260924 05:13:08.678109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 22630
I20260924 05:13:08.682585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 22705
2026-09-24T05:13:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2778 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:13:08 dist-test-slave-mr2f krb5kdc[22921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:13:08 dist-test-slave-mr2f krb5kdc[22921](info): set up 2 sockets
Sep 24 05:13:08 dist-test-slave-mr2f krb5kdc[22921](info): commencing operation
krb5kdc: starting...
W20260924 05:13:09.691023 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/test-user.keytab.
Sep 24 05:13:09 dist-test-slave-mr2f krb5kdc[22921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:13:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:13:09.797570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38387
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38387,127.0.98.61:42349,127.0.98.60:41677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:13:09.875833 22934 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:13:09.876067 22934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:09.876103 22934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:09.876145 22934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:09.877282 22934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:09.877348 22934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:09.877372 22934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:09.877393 22934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:09.877419 22934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:09.878851 22934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38387,127.0.98.61:42349,127.0.98.60:41677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:09.879117 22934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:09.879328 22934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:13:09.883807 22939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:09.883818 22942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:09.883821 22940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:09.884145 22934 server_base.cc:1034] running on GCE node
I20260924 05:13:09.884392 22934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:09.884691 22934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:09.885846 22934 hybrid_clock.cc:648] HybridClock initialized: now 1790226789885842 us; error 35 us; skew 500 ppm
Sep 24 05:13:09 dist-test-slave-mr2f krb5kdc[22921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:09.888195 22934 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:13:09.888577 22934 webserver.cc:460] Webserver started at http://127.0.98.62:34935/ using document root <none> and password file <none>
I20260924 05:13:09.888799 22934 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:09.888859 22934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:09.888995 22934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:09.889850 22934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/data/instance:
uuid: "200ba192f4ce420fbdf0a10b814feec5"
format_stamp: "Formatted at 2026-09-24 05:13:09 on dist-test-slave-mr2f"
server_key: "a130b74ece7d983a0b1236c862689d6b"
server_key_iv: "7cbda916d9d5dfe606a5104441c73caf"
server_key_version: "encryptionkey@0"
I20260924 05:13:09.890172 22934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/wal/instance:
uuid: "200ba192f4ce420fbdf0a10b814feec5"
format_stamp: "Formatted at 2026-09-24 05:13:09 on dist-test-slave-mr2f"
server_key: "a130b74ece7d983a0b1236c862689d6b"
server_key_iv: "7cbda916d9d5dfe606a5104441c73caf"
server_key_version: "encryptionkey@0"
I20260924 05:13:09.891503 22934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 05:13:09.892393 22949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:09.892550 22934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:13:09.892647 22934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/wal
uuid: "200ba192f4ce420fbdf0a10b814feec5"
format_stamp: "Formatted at 2026-09-24 05:13:09 on dist-test-slave-mr2f"
server_key: "a130b74ece7d983a0b1236c862689d6b"
server_key_iv: "7cbda916d9d5dfe606a5104441c73caf"
server_key_version: "encryptionkey@0"
I20260924 05:13:09.892728 22934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:09.908003 22934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:09.909565 22934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:09.909722 22934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:09.913367 22934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38387
I20260924 05:13:09.913395 23001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38387 every 8 connection(s)
I20260924 05:13:09.913676 22934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:13:09.914592 23002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:09.916044 23002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:13:09 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:13:09.921644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 22934
I20260924 05:13:09.921716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-0/wal/instance
I20260924 05:13:09.921898 392 external_mini_cluster.cc:1428] Setting key 8b1a9d64e457b21021381ce24842b741
I20260924 05:13:09.925171 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:09.916587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38387 (local address 127.0.0.1:42900)
0924 05:13:09.916778 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:09.916814 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:09.917242 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:09.917507 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:09.917509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:09.917734 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:09.917882 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:09.917885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:09.918876 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:09.918877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:09.919488 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:09.919489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:09.919969 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:09.919970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:09.920026 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:09.920466 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:09.920473 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:09.921885 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:09.923839 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:09.923844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:09.923846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:09.924045 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:09.924300 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:09.924300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:09.924301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:09.924373 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:09.924778 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:09.924782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:09.924833 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 05:13:09.924899 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 05:13:09.925011 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260924 05:13:09.925278 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:09.916661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.0.98.62:38387)
0924 05:13:09.916785 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:09.916793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:09.916902 (+ 109us) server_negotiation.cc:407] Connection header received
0924 05:13:09.917380 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:09.917382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:09.917406 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:09.917489 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:09.917977 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:09.918741 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:09.919587 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:09.919868 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:09.919985 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:09.921965 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:09.921972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:09.921972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:09.921978 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:09.923757 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:09.924129 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:09.924132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:09.924132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:09.924230 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:09.924436 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:09.924437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:09.924437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:09.924710 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:09.924794 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:09.924952 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 05:13:09.925113 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260924 05:13:09.925963 23002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:09.926265 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:09.926156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42349 (local address 127.0.0.1:45228)
0924 05:13:09.926197 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:09.926248 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:13:09.926425 22950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
I20260924 05:13:09.926852 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:09.926738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42349 (local address 127.0.0.1:45232)
0924 05:13:09.926808 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:09.926833 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 05:13:09.926983 23002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42349: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 05:13:09.943094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42349
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38387,127.0.98.61:42349,127.0.98.60:41677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:13:09.976190 23002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } attempt: 1
I20260924 05:13:09.976724 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:09.976573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42349 (local address 127.0.0.1:45242)
0924 05:13:09.976657 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:09.976697 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 05:13:09.977114 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:09.977033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42349 (local address 127.0.0.1:45248)
0924 05:13:09.977074 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:09.977097 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:13:09.977238 23002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42349: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
W20260924 05:13:10.021200 23009 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:13:10.021395 23009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:10.021412 23009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:10.021431 23009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:10.022462 23009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:10.022504 23009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:10.022518 23009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:10.022531 23009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:10.022543 23009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:10.023985 23009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38387,127.0.98.61:42349,127.0.98.60:41677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:10.024173 23009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:10.024324 23009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:13:10.028872 23017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:10.028882 23014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:10.028898 23009 server_base.cc:1034] running on GCE node
W20260924 05:13:10.028882 23015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:10.029305 23009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:13:10.029568 23009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:10.030710 23009 hybrid_clock.cc:648] HybridClock initialized: now 1790226790030701 us; error 31 us; skew 500 ppm
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:10.033026 23009 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 05:13:10.033459 23009 webserver.cc:460] Webserver started at http://127.0.98.61:39483/ using document root <none> and password file <none>
I20260924 05:13:10.033681 23009 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:10.033726 23009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:10.033885 23009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:10.034925 23009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/data/instance:
uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad"
format_stamp: "Formatted at 2026-09-24 05:13:10 on dist-test-slave-mr2f"
server_key: "04d3054983a576baa6ba0c8a0ecf1c36"
server_key_iv: "e4aa16e435435b3280e906951cfa0313"
server_key_version: "encryptionkey@0"
I20260924 05:13:10.035272 23009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/wal/instance:
uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad"
format_stamp: "Formatted at 2026-09-24 05:13:10 on dist-test-slave-mr2f"
server_key: "04d3054983a576baa6ba0c8a0ecf1c36"
server_key_iv: "e4aa16e435435b3280e906951cfa0313"
server_key_version: "encryptionkey@0"
I20260924 05:13:10.036533 23009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:13:10.037189 23024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:10.037348 23009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:13:10.037441 23009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/wal
uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad"
format_stamp: "Formatted at 2026-09-24 05:13:10 on dist-test-slave-mr2f"
server_key: "04d3054983a576baa6ba0c8a0ecf1c36"
server_key_iv: "e4aa16e435435b3280e906951cfa0313"
server_key_version: "encryptionkey@0"
I20260924 05:13:10.037518 23009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:10.045382 23002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } attempt: 2
I20260924 05:13:10.045737 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.045621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42349 (local address 127.0.0.1:45250)
0924 05:13:10.045676 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.045711 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 05:13:10.046030 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.045934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42349 (local address 127.0.0.1:45264)
0924 05:13:10.045982 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.046005 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 05:13:10.046113 23002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42349: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42349: connect: Connection refused (error 111)
I20260924 05:13:10.059041 23009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:10.060674 23009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:10.060832 23009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:10.064823 23009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42349
I20260924 05:13:10.064854 23076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42349 every 8 connection(s)
I20260924 05:13:10.065186 23009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:13:10.066089 23077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:10.067209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 23009
I20260924 05:13:10.067291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-1/wal/instance
I20260924 05:13:10.067493 392 external_mini_cluster.cc:1428] Setting key 2ef92f63a98f5c908c9026a024e5361c
I20260924 05:13:10.067734 23077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 05:13:10.076628 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.068370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.0.98.62:38387)
0924 05:13:10.068429 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.068432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.068649 (+ 217us) server_negotiation.cc:407] Connection header received
0924 05:13:10.069262 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.069263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.069279 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.069335 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.069905 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.070385 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.071383 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.071584 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.071667 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.073831 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.073833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.073833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.073841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.075388 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.075724 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.075724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.075725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.075772 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.075997 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.075998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.075998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.076119 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.076169 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.076413 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.076498 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:13:10.076702 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.068299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38387 (local address 127.0.0.1:42906)
0924 05:13:10.068534 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.068560 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:10.069151 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:10.069403 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:10.069406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:10.069639 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:10.069814 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.069817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.070490 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.070491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.071284 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.071286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.071689 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.071690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.071812 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.072292 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:10.072300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:10.073738 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:10.075456 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.075462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.075464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.075661 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.075828 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.075828 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.075828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.075920 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.076176 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:10.076181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:10.076286 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 05:13:10.076366 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 05:13:10.076523 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":69,"threads_started":1}
I20260924 05:13:10.077517 23077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:13:10.084378 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.077751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42349 (local address 127.0.0.1:45276)
0924 05:13:10.077934 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.077945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:10.077982 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:10.078114 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:10.078115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:10.078262 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:10.078317 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.078318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.079138 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.079139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.079670 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.079671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.080147 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.080148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.080211 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.080585 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:10.080590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:10.081654 (+ 1064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:10.083363 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.083364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.083364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.083541 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.083752 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.083753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.083754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.083806 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.084083 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:10.084083 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:10.084118 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 05:13:10.084182 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 05:13:10.084270 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 05:13:10.084450 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.077818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.0.98.61:42349)
0924 05:13:10.077964 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.077969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.077992 (+ 23us) server_negotiation.cc:407] Connection header received
0924 05:13:10.078035 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.078036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.078047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.078103 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.078380 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.079074 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.079766 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.080059 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.080147 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.081736 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.081743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.081744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.081750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.083266 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.083630 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.083633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.083633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.083688 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.083876 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.083877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.083877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.084012 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.084102 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.084207 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.084320 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260924 05:13:10.084936 23077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:10.085321 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.085195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38392)
0924 05:13:10.085250 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.085297 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 05:13:10.085498 23026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
I20260924 05:13:10.085922 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.085817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38400)
0924 05:13:10.085863 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.085893 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 05:13:10.086021 23077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41677: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 05:13:10.090435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhdUPbm/build/release/bin/kudu
/tmp/dist-test-taskhdUPbm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41677
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38387,127.0.98.61:42349,127.0.98.60:41677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:13:10.135195 23077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } attempt: 1
I20260924 05:13:10.135667 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.135548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38410)
0924 05:13:10.135606 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.135642 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 05:13:10.136009 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.135934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38424)
0924 05:13:10.135971 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.135995 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:13:10.136107 23077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41677: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
I20260924 05:13:10.137238 23002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } attempt: 3
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:13:10.143824 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.137533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.0.98.61:42349)
0924 05:13:10.137594 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.137596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.137607 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:13:10.137654 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.137654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.137665 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.137720 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.138028 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.138516 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.139269 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.139500 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.139576 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.141452 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.141454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.141454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.141461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.142736 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.143094 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.143095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.143095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.143151 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.143335 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.143336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.143337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.143455 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.143527 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.143675 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.143742 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:13:10.143829 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.137480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42349 (local address 127.0.0.1:45286)
0924 05:13:10.137530 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.137541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:10.137595 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:10.137725 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:10.137725 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:10.137875 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:10.137933 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.137934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.138605 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.138606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.139186 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.139188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.139582 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.139583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.139659 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.140134 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:10.140139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:10.141367 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:10.142854 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.142855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.142856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.143023 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.143214 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.143215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.143215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.143271 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.143529 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:10.143530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:10.143571 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 05:13:10.143652 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 05:13:10.143741 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:13:10.144318 23002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:10.144606 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.144533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38426)
0924 05:13:10.144567 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.144591 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:13:10.144855 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.144797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38428)
0924 05:13:10.144825 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.144844 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 05:13:10.144914 23002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41677: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
W20260924 05:13:10.168138 23084 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:13:10.168345 23084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:13:10.168382 23084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:13:10.168424 23084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:13:10.169548 23084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:13:10.169600 23084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:13:10.169621 23084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:13:10.169641 23084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:13:10.169657 23084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:13:10.171573 23084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38387,127.0.98.61:42349,127.0.98.60:41677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:13:10.171747 23084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:13:10.171897 23084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:13:10.176030 23002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } attempt: 1
W20260924 05:13:10.176422 23092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:10.176510 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.176368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38440)
0924 05:13:10.176445 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.176473 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 05:13:10.176473 23084 server_base.cc:1034] running on GCE node
W20260924 05:13:10.176432 23089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:13:10.176699 23090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:13:10.176900 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.176801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38452)
0924 05:13:10.176838 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.176867 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 05:13:10.176939 23084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260924 05:13:10.177016 23002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41677: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41677: connect: Connection refused (error 111)
I20260924 05:13:10.177246 23084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:13:10.178414 23084 hybrid_clock.cc:648] HybridClock initialized: now 1790226790178402 us; error 23 us; skew 500 ppm
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:13:10.180838 23084 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 05:13:10.181231 23084 webserver.cc:460] Webserver started at http://127.0.98.60:41871/ using document root <none> and password file <none>
I20260924 05:13:10.181449 23084 fs_manager.cc:359] Metadata directory not provided
I20260924 05:13:10.181514 23084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:13:10.181659 23084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:13:10.182657 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/data/instance:
uuid: "de0dad0df8d14b109fc77f4e909e5ed2"
format_stamp: "Formatted at 2026-09-24 05:13:10 on dist-test-slave-mr2f"
server_key: "21944e35c6730d0d4e9560723820543b"
server_key_iv: "5e56b459abce496f8ec86c77e3d105a3"
server_key_version: "encryptionkey@0"
I20260924 05:13:10.183069 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/wal/instance:
uuid: "de0dad0df8d14b109fc77f4e909e5ed2"
format_stamp: "Formatted at 2026-09-24 05:13:10 on dist-test-slave-mr2f"
server_key: "21944e35c6730d0d4e9560723820543b"
server_key_iv: "5e56b459abce496f8ec86c77e3d105a3"
server_key_version: "encryptionkey@0"
I20260924 05:13:10.184340 23084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:13:10.185029 23099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:13:10.185199 23084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:13:10.185295 23084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/wal
uuid: "de0dad0df8d14b109fc77f4e909e5ed2"
format_stamp: "Formatted at 2026-09-24 05:13:10 on dist-test-slave-mr2f"
server_key: "21944e35c6730d0d4e9560723820543b"
server_key_iv: "5e56b459abce496f8ec86c77e3d105a3"
server_key_version: "encryptionkey@0"
I20260924 05:13:10.185379 23084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:13:10.195247 23084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:13:10.197415 23084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:13:10.197568 23084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:13:10.201542 23084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41677
I20260924 05:13:10.201570 23151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41677 every 8 connection(s)
I20260924 05:13:10.201870 23084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:13:10.202850 23152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:13:10.204263 23077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } attempt: 2
I20260924 05:13:10.204174 23152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:10.204401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhdUPbm/build/release/bin/kudu as pid 23084
I20260924 05:13:10.204490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhdUPbm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790226527740525-392-0/minicluster-data/master-2/wal/instance
I20260924 05:13:10.204705 392 external_mini_cluster.cc:1428] Setting key 0bbe641fec59272764bf4a58120a7e11
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:13:10.214545 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.204986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.0.98.62:38387)
0924 05:13:10.205026 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.205028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.205201 (+ 173us) server_negotiation.cc:407] Connection header received
0924 05:13:10.205549 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.205550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.205562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.205613 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.206278 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.206975 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.207840 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.208070 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.208131 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.210534 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.210535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.210536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.210543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.212875 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.213454 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.213455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.213456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.213524 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.213787 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.213788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.213789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.213924 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.213995 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.214335 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.214420 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:13:10.214601 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.204542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38462)
0924 05:13:10.204590 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.204602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:10.204658 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:10.205594 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:10.205596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:10.205753 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:10.205791 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.205793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.207168 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.207169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.207929 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.207930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.208443 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.208444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.208798 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.209282 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:10.209288 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:10.211055 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:10.213510 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.213511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.213511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.213726 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.213949 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.213949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.213950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.214021 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.214291 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:10.214292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:10.214338 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:13:10.214407 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 05:13:10.214502 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 05:13:10.215139 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.204661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.0.98.60:41677)
0924 05:13:10.204817 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.204826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.204858 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:13:10.205444 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.205452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.205482 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.205594 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.205886 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.207095 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.208014 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.208362 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.208441 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.211140 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.211147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.211148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.211155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.213457 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.213808 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.213810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.213810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.213864 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.214086 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.214087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.214087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.214219 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.214307 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.214447 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.214569 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260924 05:13:10.215235 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.204936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38387 (local address 127.0.0.1:42916)
0924 05:13:10.205084 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.205123 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:10.205444 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:10.205692 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:10.205695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:10.206042 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:10.206172 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.206177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.207052 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.207053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.207728 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.207730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.208195 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.208196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.208321 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.208821 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:10.208831 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:10.210436 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:10.212966 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.212972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.212974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.213370 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.213591 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.213592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.213592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.213713 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.214022 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:10.214029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:10.214178 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 05:13:10.214632 (+ 454us) client_negotiation.cc:240] Negotiation successful
0924 05:13:10.214752 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260924 05:13:10.215430 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.205142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.0.98.60:41677)
0924 05:13:10.205369 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.205370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.205376 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:13:10.205458 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.205459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.205483 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.206306 (+ 823us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.206331 (+ 25us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.207097 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.208133 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.208447 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.208568 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.211679 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.211680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.211680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.211687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.213886 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.214299 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.214300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.214300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.214352 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.214551 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.214551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.214552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.214668 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.215181 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.215219 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.215306 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":238,"server-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260924 05:13:10.216048 23152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:13:10.217187 23077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad: Bootstrap starting.
I20260924 05:13:10.218114 23077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:10.218453 23077 log.cc:824] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:10.219244 23077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad: No bootstrap required, opened a new log
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:13:10.221211 23077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } }
I20260924 05:13:10.221331 23077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c87d2e95e4f4a70b0bed29e8adce2ad, State: Initialized, Role: FOLLOWER
I20260924 05:13:10.221402 23077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } }
I20260924 05:13:10.221706 23077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:10.221992 23159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.222160 23159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:10.222994 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.216318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.0.98.61:42349)
0924 05:13:10.216364 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.216366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.216408 (+ 42us) server_negotiation.cc:407] Connection header received
0924 05:13:10.216484 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.216484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.216491 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.216537 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.216775 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.217193 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.218112 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.218324 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.218371 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.220124 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.220125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.220126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.220132 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.221594 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.221987 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.221988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.221988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.222076 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.222293 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.222295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.222295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.222480 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.222540 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.222789 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.222883 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:13:10.223367 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.216259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42349 (local address 127.0.0.1:45298)
0924 05:13:10.216361 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.216371 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:10.216421 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:10.216540 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:10.216540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:10.216667 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:10.216713 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.216714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.217265 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.217266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.218023 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.218025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.218388 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.218389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.218481 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.218974 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:10.218978 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:10.220036 (+ 1058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:10.221675 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.221675 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.221676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.221895 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.222150 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.222151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.222152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.222212 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.222610 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:10.222611 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:10.222657 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:13:10.223106 (+ 449us) client_negotiation.cc:240] Negotiation successful
0924 05:13:10.223269 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:13:10.223740 23152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } has no permanent_uuid. Determining permanent_uuid...
W20260924 05:13:10.224869 23170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:13:10.224925 23170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:13:10.224450 23009 master_runner.cc:186] Error getting master registration for 127.0.98.62:38387: OK, instance_id { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" instance_seqno: 1790226789912348 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:13:10.227151 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.216119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.0.98.62:38387)
0924 05:13:10.216184 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.216186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.218800 (+ 2614us) server_negotiation.cc:407] Connection header received
0924 05:13:10.218859 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.218860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.218869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.218914 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.219456 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.220083 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.221122 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.221376 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.221448 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.224092 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.224093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.224094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.224100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.225791 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.226143 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.226144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.226144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.226193 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.226368 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.226369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.226369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.226475 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.226595 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.226999 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.227068 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 05:13:10.227226 23009 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:13:10.232311 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.224100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38476)
0924 05:13:10.224161 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.224169 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:10.224202 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:10.224553 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:10.224554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:10.224711 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:10.224738 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.224740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.226094 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.226094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.226855 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.226857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.227289 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.227289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.227336 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.227727 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:10.227733 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:10.229099 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:10.231250 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.231251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.231252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.231462 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.231672 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.231673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.231673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.231750 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.232058 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:10.232059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:10.232092 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 05:13:10.232133 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 05:13:10.232220 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 05:13:10.232611 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.224172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.0.98.60:41677)
0924 05:13:10.224206 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.224208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.224219 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:13:10.224242 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.224243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.224251 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.224304 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.224823 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.226015 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.226934 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.227205 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.227473 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.229183 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.229184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.229184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.229191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.231168 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.231544 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.231545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.231545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.231596 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.231820 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.231820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.231821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.231975 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.232412 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.232450 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.232529 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:13:10.234220 23152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2: Bootstrap starting.
I20260924 05:13:10.234941 23152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:10.235260 23152 log.cc:824] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:10.235451 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.227632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.0.98.61:42349)
0924 05:13:10.227663 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.227665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.227672 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:13:10.227724 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.227725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.227735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.227778 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.228378 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.228809 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.229606 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.229797 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.229877 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.232528 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.232529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.232530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.232535 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.233937 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.234385 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.234386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.234386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.234439 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.234631 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.234632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.234632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.234743 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.234883 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.235266 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.235365 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:13:10.236169 23152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2: No bootstrap required, opened a new log
I20260924 05:13:10.238059 23152 raft_consensus.cc:348] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } }
I20260924 05:13:10.238181 23152 raft_consensus.cc:729] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de0dad0df8d14b109fc77f4e909e5ed2, State: Initialized, Role: FOLLOWER
I20260924 05:13:10.238250 23152 consensus_queue.cc:260] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } }
I20260924 05:13:10.238520 23152 sys_catalog.cc:564] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:10.238502 23173 sys_catalog.cc:455] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.238696 23173 sys_catalog.cc:458] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:10.239187 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:13:10.240414 22976 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "200ba192f4ce420fbdf0a10b814feec5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42924
W20260924 05:13:10.241864 23184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:13:10.241907 23184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:13:10.240846 23084 master_runner.cc:186] Error getting master registration for 127.0.98.62:38387: OK, instance_id { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" instance_seqno: 1790226789912348 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:13:10.242753 23084 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:13:10.252151 23002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } attempt: 2
Sep 24 05:13:10 dist-test-slave-mr2f krb5kdc[22921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:13:10.258826 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.252351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41677 (local address 127.0.0.1:38484)
0924 05:13:10.252391 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:13:10.252401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:13:10.252462 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:13:10.252609 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:13:10.252610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:13:10.252733 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:13:10.252764 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.252766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.253538 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.253556 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.254075 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:13:10.254076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.254506 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.254507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:13:10.254563 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.255006 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:13:10.255011 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:13:10.256296 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:13:10.257925 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.257926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.257926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.258081 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.258248 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:13:10.258249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:13:10.258249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:13:10.258294 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:13:10.258526 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:13:10.258527 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:13:10.258579 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 05:13:10.258637 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:13:10.258722 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:13:10.258834 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.252408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.0.98.60:41677)
0924 05:13:10.252456 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.252458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.252466 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:13:10.252529 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.252529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.252538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.252595 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.252844 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.253477 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.254167 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.254446 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.254508 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.256377 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.256378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.256379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.256385 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.257846 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.258149 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.258150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.258150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.258199 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.258365 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.258366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.258366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.258479 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.258534 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.258658 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.258722 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:13:10.260366 23002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5: Bootstrap starting.
I20260924 05:13:10.261027 23002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5: Neither blocks nor log segments found. Creating new log.
I20260924 05:13:10.261315 23002 log.cc:824] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:13:10.262090 23002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5: No bootstrap required, opened a new log
I20260924 05:13:10.263386 23002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } }
I20260924 05:13:10.263516 23002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200ba192f4ce420fbdf0a10b814feec5, State: Initialized, Role: FOLLOWER
I20260924 05:13:10.263599 23002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } }
I20260924 05:13:10.263890 23002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:13:10.263985 22976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 05:13:10.264034 22976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:10.264861 22976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } }
I20260924 05:13:10.265129 22976 leader_election.cc:288] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [CANDIDATE]: Term 1 election: Requested vote from peers 3c87d2e95e4f4a70b0bed29e8adce2ad (127.0.98.61:42349), de0dad0df8d14b109fc77f4e909e5ed2 (127.0.98.60:41677)
I20260924 05:13:10.266057 23051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "200ba192f4ce420fbdf0a10b814feec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad"
I20260924 05:13:10.266173 23051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:10.266894 23126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "200ba192f4ce420fbdf0a10b814feec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "de0dad0df8d14b109fc77f4e909e5ed2"
I20260924 05:13:10.267019 23126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:13:10.267835 23126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 200ba192f4ce420fbdf0a10b814feec5 in term 1.
I20260924 05:13:10.269336 22951 leader_election.cc:302] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200ba192f4ce420fbdf0a10b814feec5, de0dad0df8d14b109fc77f4e909e5ed2; no voters:
I20260924 05:13:10.270311 23051 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 200ba192f4ce420fbdf0a10b814feec5 in term 1.
I20260924 05:13:10.272018 22934 master_runner.cc:263] No leader master found from master 200ba192f4ce420fbdf0a10b814feec5
I20260924 05:13:10.272070 22934 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 05:13:10.266215 23198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:13:10.272295 23198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:13:10.263844 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.272450 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:10.272544 23187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:13:10.272641 23187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [term 1 LEADER]: Becoming Leader. State: Replica: 200ba192f4ce420fbdf0a10b814feec5, State: Running, Role: LEADER
I20260924 05:13:10.272749 23187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } }
I20260924 05:13:10.273209 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200ba192f4ce420fbdf0a10b814feec5. Latest consensus state: current_term: 1 leader_uuid: "200ba192f4ce420fbdf0a10b814feec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.273348 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:10.274664 23206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:13:10.275327 23206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:13:10.283816 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.273719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.0.98.60:41677)
0924 05:13:10.273769 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.273771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.276149 (+ 2378us) server_negotiation.cc:407] Connection header received
0924 05:13:10.276246 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.276247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.276256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.276286 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.277048 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.277767 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.278889 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.279186 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.279230 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.280913 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.280914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.280915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.280921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.282638 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.283032 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.283033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.283033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.283086 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.283257 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.283258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.283258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.283390 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.283479 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.283661 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.283729 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:13:10.290910 23126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [term 1 FOLLOWER]: Refusing update from remote peer 200ba192f4ce420fbdf0a10b814feec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:13:10.291262 23204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:13:10.292935 23051 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [term 1 FOLLOWER]: Refusing update from remote peer 200ba192f4ce420fbdf0a10b814feec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:13:10.293144 23173 sys_catalog.cc:455] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200ba192f4ce420fbdf0a10b814feec5. Latest consensus state: current_term: 1 leader_uuid: "200ba192f4ce420fbdf0a10b814feec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.293224 23173 sys_catalog.cc:458] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:10.293308 23204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:13:10.294471 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.274408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.0.98.61:42349)
0924 05:13:10.274444 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.274446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.280125 (+ 5679us) server_negotiation.cc:407] Connection header received
0924 05:13:10.282048 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.282068 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.282079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.282110 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.287308 (+ 5198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.288059 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.289374 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.289681 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.289724 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.291053 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.291054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.291054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.291061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.292860 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.293604 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.293605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.293605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.293657 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.293846 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.293847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.293847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.293987 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.294066 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.294275 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.294363 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:13:10.295372 23159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200ba192f4ce420fbdf0a10b814feec5. Latest consensus state: current_term: 1 leader_uuid: "200ba192f4ce420fbdf0a10b814feec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.295451 23159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:10.299998 23204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "200ba192f4ce420fbdf0a10b814feec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.300091 23204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:10.300586 23159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "200ba192f4ce420fbdf0a10b814feec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.300685 23159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:10.300958 23210 mvcc.cc:205] Tried to move back new op lower bound from 7332768933028884480 to 7332768932958228480. Current Snapshot: MvccSnapshot[applied={T|T < 7332768933028884480}]
I20260924 05:13:10.301105 23213 mvcc.cc:205] Tried to move back new op lower bound from 7332768933028884480 to 7332768932958228480. Current Snapshot: MvccSnapshot[applied={T|T < 7332768933028884480}]
I20260924 05:13:10.301769 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:10.274079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.0.98.62:38387)
0924 05:13:10.274114 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.274115 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.278888 (+ 4773us) server_negotiation.cc:407] Connection header received
0924 05:13:10.278987 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.278988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.278997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.279024 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.280315 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.281026 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.286896 (+ 5870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:10.287199 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:10.287237 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:10.298860 (+ 11623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:13:10.298861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:13:10.298861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:13:10.298868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:13:10.300544 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.300935 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.300936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.300936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.301014 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:13:10.301207 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:13:10.301208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:13:10.301208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:13:10.301328 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:13:10.301413 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:10.301618 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 05:13:10.301689 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:13:10.303248 23206 catalog_manager.cc:1347] Generated new cluster ID: b427555e2c2b48f686d3e70aeed2140a
I20260924 05:13:10.303292 23206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:13:10.314736 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "200ba192f4ce420fbdf0a10b814feec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.314836 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:13:10.315320 23173 sys_catalog.cc:455] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "200ba192f4ce420fbdf0a10b814feec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "200ba192f4ce420fbdf0a10b814feec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38387 } } peers { permanent_uuid: "3c87d2e95e4f4a70b0bed29e8adce2ad" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42349 } } peers { permanent_uuid: "de0dad0df8d14b109fc77f4e909e5ed2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41677 } } }
I20260924 05:13:10.315393 23173 sys_catalog.cc:458] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:13:10.315667 23211 mvcc.cc:205] Tried to move back new op lower bound from 7332768933028884480 to 7332768932958228480. Current Snapshot: MvccSnapshot[applied={T|T < 7332768933028884480}]
I20260924 05:13:10.324983 23206 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:13:10.325714 23206 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:13:10.340631 23206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 200ba192f4ce420fbdf0a10b814feec5: Generated new TSK 0
I20260924 05:13:10.340854 23206 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:13:10.361680 23005 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.360423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.0.98.62:38387)
0924 05:13:10.360463 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.360466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.361066 (+ 600us) server_negotiation.cc:407] Connection header received
0924 05:13:10.361169 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.361170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.361181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.361210 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.361617 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42930: BlockingRecv error: recv got EOF from 127.0.0.1:42930 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:13:10.362425 23157 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.360796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.0.98.60:41677)
0924 05:13:10.360845 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.360847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.361910 (+ 1063us) server_negotiation.cc:407] Connection header received
0924 05:13:10.362002 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.362003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.362012 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.362040 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.362364 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38492: BlockingRecv error: recv got EOF from 127.0.0.1:38492 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 05:13:10.363533 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.362793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.0.98.61:42349)
0924 05:13:10.362848 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.362850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.362983 (+ 133us) server_negotiation.cc:407] Connection header received
0924 05:13:10.363084 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.363085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.363101 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.363131 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.363464 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45332: BlockingRecv error: recv got EOF from 127.0.0.1:45332 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 05:13:10.365638 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.364950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.0.98.61:42349)
0924 05:13:10.364984 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.364986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.365152 (+ 166us) server_negotiation.cc:407] Connection header received
0924 05:13:10.365252 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.365253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.365269 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.365299 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.365595 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45336: BlockingRecv error: recv got EOF from 127.0.0.1:45336 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:13:10.368505 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.367777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.0.98.61:42349)
0924 05:13:10.367820 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.367822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.368046 (+ 224us) server_negotiation.cc:407] Connection header received
0924 05:13:10.368139 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.368139 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.368149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.368177 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.368465 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45346: BlockingRecv error: recv got EOF from 127.0.0.1:45346 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:13:10.372233 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.371541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.0.98.61:42349)
0924 05:13:10.371572 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.371574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.371761 (+ 187us) server_negotiation.cc:407] Connection header received
0924 05:13:10.371872 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.371873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.371882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.371910 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.372195 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45356: BlockingRecv error: recv got EOF from 127.0.0.1:45356 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:13:10.378041 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.377323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.0.98.61:42349)
0924 05:13:10.377356 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.377358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.377540 (+ 182us) server_negotiation.cc:407] Connection header received
0924 05:13:10.377629 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.377630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.377639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.377668 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.378002 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45362: BlockingRecv error: recv got EOF from 127.0.0.1:45362 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 05:13:10.387944 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.387172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45374 (local address 127.0.98.61:42349)
0924 05:13:10.387206 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.387207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.387412 (+ 205us) server_negotiation.cc:407] Connection header received
0924 05:13:10.387502 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.387503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.387514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.387541 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.387903 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45374: BlockingRecv error: recv got EOF from 127.0.0.1:45374 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:13:10.406126 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.405272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.0.98.61:42349)
0924 05:13:10.405314 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.405316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.405544 (+ 228us) server_negotiation.cc:407] Connection header received
0924 05:13:10.405637 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.405639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.405649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.405678 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.406066 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45390: BlockingRecv error: recv got EOF from 127.0.0.1:45390 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 05:13:10.440486 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.439507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.0.98.61:42349)
0924 05:13:10.439563 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.439565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.439743 (+ 178us) server_negotiation.cc:407] Connection header received
0924 05:13:10.439850 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.439851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.439861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.439914 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.440407 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45402: BlockingRecv error: recv got EOF from 127.0.0.1:45402 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 05:13:10.506302 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.505682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.0.98.61:42349)
0924 05:13:10.505731 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.505733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.505819 (+ 86us) server_negotiation.cc:407] Connection header received
0924 05:13:10.505868 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.505869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.505880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.505920 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.506244 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45408: BlockingRecv error: recv got EOF from 127.0.0.1:45408 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 05:13:10.636135 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.635450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.0.98.61:42349)
0924 05:13:10.635518 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.635520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.635651 (+ 131us) server_negotiation.cc:407] Connection header received
0924 05:13:10.635698 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.635699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.635710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.635761 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.636089 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45418: BlockingRecv error: recv got EOF from 127.0.0.1:45418 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 05:13:10.893982 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:13:10.893369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.0.98.61:42349)
0924 05:13:10.893433 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:10.893435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:10.893496 (+ 61us) server_negotiation.cc:407] Connection header received
0924 05:13:10.893548 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:10.893549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:10.893559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:10.893611 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:13:10.893927 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45432: BlockingRecv error: recv got EOF from 127.0.0.1:45432 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:13:11.225349 23170 catalog_manager.cc:1259] Loaded cluster ID: b427555e2c2b48f686d3e70aeed2140a
I20260924 05:13:11.225425 23170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad: loading cluster ID for follower catalog manager: success
I20260924 05:13:11.226224 23170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad: acquiring CA information for follower catalog manager: success
I20260924 05:13:11.226603 23170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c87d2e95e4f4a70b0bed29e8adce2ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:13:11.242275 23184 catalog_manager.cc:1259] Loaded cluster ID: b427555e2c2b48f686d3e70aeed2140a
I20260924 05:13:11.242336 23184 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2: loading cluster ID for follower catalog manager: success
I20260924 05:13:11.243170 23184 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2: acquiring CA information for follower catalog manager: success
I20260924 05:13:11.243467 23184 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de0dad0df8d14b109fc77f4e909e5ed2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:13:11.409457 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:11.407246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.0.98.61:42349)
0924 05:13:11.407346 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:11.407348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:11.407358 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:13:11.407436 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:11.407437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:11.407481 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:11.407557 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:11.407692 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.408140 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.408796 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.408977 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.409021 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:11.409149 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:11.409245 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:11.409295 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:11.409331 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 05:13:11.409346 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":59,"threads_started":1}
I20260924 05:13:11.413319 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:11.411265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.0.98.60:41677)
0924 05:13:11.411361 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:11.411363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:11.411371 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:13:11.411415 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:11.411415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:11.411451 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:11.411515 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:11.411632 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.412072 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.412699 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.412873 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.412921 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:11.413029 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:11.413114 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:11.413163 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:11.413196 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 05:13:11.413210 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":51,"threads_started":1}
I20260924 05:13:11.417639 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:11.415077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.0.98.60:41677)
0924 05:13:11.415132 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:11.415133 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:11.415139 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:13:11.415158 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:11.415158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:11.415166 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:11.415211 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:11.415364 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.415980 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.416857 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.417161 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.417223 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:11.417387 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:11.417419 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:11.417469 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:11.417541 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 05:13:11.417564 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:13:11.418304 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:11.414876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.0.98.62:38387)
0924 05:13:11.415123 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:11.415125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:11.415132 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:13:11.415178 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:11.415179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:11.415245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:11.415342 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:11.415510 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.416309 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.417201 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.417448 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.417511 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:11.417900 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:11.418024 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:11.418070 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:11.418187 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 05:13:11.418210 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":70,"threads_started":1}
I20260924 05:13:11.418308 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:13:11.414892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.0.98.61:42349)
0924 05:13:11.415025 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 05:13:11.415027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:13:11.415551 (+ 524us) server_negotiation.cc:407] Connection header received
0924 05:13:11.415577 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:13:11.415578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:13:11.415587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:13:11.415635 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:13:11.415776 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.416465 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.417415 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:13:11.417768 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:13:11.417823 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:13:11.417996 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:13:11.418029 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:13:11.418075 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:13:11.418138 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 05:13:11.418157 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 05:13:11.425822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 22934
I20260924 05:13:11.431013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 23009
I20260924 05:13:11.435765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhdUPbm/build/release/bin/kudu with pid 23084
2026-09-24T05:13:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2752 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5531 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (263701 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST